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ABSTRACT 


During  the  late  90s,  NASA  retrofitted  the  Space  Shuttle  fleet  with  a  “glass 
cockpit.”  The  new  displays  replicated  legacy  formats  developed  in  the  70s,  and 
did  not  leverage  20  years  of  display  technology  and  human  factors  advances.  To 
address  this  shortcoming  the  Cockpit  Avionics  Upgrade  (CAU)  was  initiated  to 
reduce  mental  workload  (MW),  increase  situational  awareness  (SA),  and 
enhance  performance.  Despite  the  CAU  demonstrating  improvements  in  MW, 
SA,  and  performance,  it  was  cancelled.  Consequently,  recorded  astronaut  data 
from  using  the  baseline  and  CAU  cockpit  configurations  was  never  tied  back  to 
cockpit  design.  This  study  assesses  the  CAU  design  employing  human  factors 
principles,  evaluates  baseline  and  CAU  simulation  data,  and  traces  MW  and  SA 
differences  back  to  CAU  design  modifications.  Significant  improvements  were 
found  in  all  measures  and  across  all  conditions.  These  improvements  were  found 
to  be  greater  for  ascent  scenarios  than  for  entry.  From  the  findings, 
recommendations  for  the  design  and  evaluation  of  future  spacecraft  cockpits  are 
made. 
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EXECUTIVE  SUMMARY 


When  the  Space  Shuttle  was  originally  developed  in  the  1970s,  the  cockpit  was 
based  on  the  latest  technology  at  the  time  including  fly  by  wire  controls  and 
computer  displays.  As  that  technology  became  obsolete,  NASA  retrofitted  the 
Space  Shuttle  fleet  with  a  modern  “glass  cockpit.”  At  first  the  new  displays 
replicated  legacy  formats,  and  did  not  leverage  nearly  30  years  of  advances  in 
display  technology  and  human  factors  design.  To  address  this  shortcoming,  the 
Cockpit  Avionics  Upgrade  (CAU)  project  was  initiated  to  update  display  formats 
to  reduce  mental  workload  (MW),  increase  situational  awareness  (SA),  and 
enhance  performance.  Despite  demonstrating  improvement  in  terms  of  MW,  SA, 
and  performance  over  the  baseline  design,  the  CAU  project  was  cancelled  in 
2004  due  to  budgetary  constraints.  Consequently,  astronaut  performance  data 
using  the  baseline  and  CAU  configurations  was  not  completely  analyzed  and  tied 
back  to  the  adopted  design  modifications.  The  purpose  of  this  study  was  to 
critique  the  CAU  design  employing  human  factors  principles,  evaluate  baseline 
and  CAU  MW  and  SA  data,  and  trace  back  specific  design  modifications  that 
contributed  to  improvements  in  MW  and  SA. 

A  literature  review  was  conducted  to  identify  how  human  factors  principles 
can  be  used  to  effectively  organize  and  present  information  in  a  design  concept 
for  a  cockpit.  It  included  consideration  of  relevant  methods  and  criteria  to 
objectively  critique  a  design  concept,  including  task  analysis,  display  design 
principles,  design  eye  position,  and  display-control  compatibility.  This  was 
followed  by  an  examination  of  the  theories  and  methods  used  in  design 
assessment,  including  measures  of  mental  workload  and  situation  awareness. 
The  information  reviewed  includes  scientific  journals,  textbooks,  NASA 
documents,  technical  standards,  and  related  prior  work. 

Based  on  the  literature  review,  a  process  was  established  to  critically 
evaluate  a  cockpit  design  change  and  assessing  its  effectiveness.  The  evaluation 
began  with  a  task  analysis.  This  was  followed  by  a  task-oriented  evaluation  of 
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the  layout  of  the  displays  from  the  operator  perspective,  the  formatting  of 
information  on  each  display,  the  hardware  attributes,  and  the  display-control 
relationship.  The  design  critique  noted  that  the  changes  in  the  use  of  color,  the 
consolidation  of  information,  and  the  use  of  pictorial  representations  of  systems 
are  relevant  to  all  crew  positions  and  in  all  phases  of  flight.  The  critique  also 
identified  the  predictive  aiding  features  of  the  ascent  horizontal  situation  display 
as  the  likely  cause  of  the  variation  between  ascent  and  entry  scenarios. 

To  assess  the  effectiveness  of  the  CAU  design  changes  over  the  baseline 
design,  MW  and  SA  data  were  analyzed.  For  every  MW  and  SA  measure,  there 
was  a  significant  improvement  favoring  the  CAU  design  vs.  the  baseline  for  all 
crew  positions  and  in  all  phases  of  flight.  The  analysis  also  revealed  that  these 
improvements  were  significantly  greater  for  ascent  operations  than  for  entry. 

It  is  recommended  that  future  manned  spacecraft  cockpits  incorporate  the 
design  principles  demonstrated  in  the  CAU,  including  the  use  of  a  consistent 
color  scheme,  task-centric  consolidation  of  information,  and  the  use  of  pictorial 
representations  of  complex  systems.  It  is  further  suggested  that  predictive  aiding 
displays  should  be  developed  wherever  possible  for  use  during  all  dynamic 
phases  of  flight.  Finally,  it  is  recommended  that  the  evaluation  of  these  future 
systems  should  include  a  mechanism  for  gathering  operator  feedback  on  the 
relative  usefulness  of  individual  design  principles. 
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I.  INTRODUCTION 


A.  BACKGROUND 

With  the  Space  Shuttle  program  ending  in  2011,  the  United  States  no 
longer  possesses  the  capability  to  launch  astronauts  into  space.  Many  follow  on 
programs  were  proposed  over  the  years,  but  none  have  been  realized  to  date 
(Frank,  2010).  The  Orion  Multi-Purpose  Crew  Vehicle  is  currently  being 
developed  as  NASA's  next  generation  manned  spacecraft,  but  it  is  not  expected 
to  be  operational  until  at  least  2021  (Stanfield,  2012).  This  gap  will  be  the 
longest  the  United  States  has  gone  without  possessing  its  own  launch  capability 
since  the  1961  dawn  of  manned  spaceflight.  It  is  inevitable  that  manned 
spaceflight  will  resume  in  the  United  States,  but  until  then  it  is  imperative  that  we 
maintain  our  technological  expertise,  and  continue  to  build  on  the  lessons 
learned  in  past  programs. 

When  the  Space  Shuttle  was  originally  developed  in  the  1970s,  the 
cockpit  was  based  on  the  latest  technology  available  at  the  time.  The  avionics 
suite  incorporated  technologies  such  as  fly  by  wire  controls  and  computer 
displays,  which  were  considered  quite  advanced  at  the  time  (McCandless,  2004). 
The  two  decades  that  followed  saw  the  development  of  glass  cockpit  technology 
in  the  commercial  sector,  driven  by  the  ever-increasing  complexity  of  air  transport 
operations.  Despite  the  fact  that  NASA  was  very  much  involved  in  this  effort,  the 
Space  Shuttle  cockpit  remained  unchanged  for  this  period  (Tranthien,  1995). 

While  the  shuttle's  original  cockpit  equipment  was  extremely  reliable,  it 
was  nonetheless  costly  to  maintain  over  time  (McCandless,  2004).  In  2000,  the 
original  electro-mechanical  cockpit  instrumentation  was  replaced  with  the 
Multifunction  Electronic  Display  System  (MEDS)  glass  cockpit  (McCandless  et 
al. ,  2005).  The  MEDS  cockpit  upgraded  all  of  the  hardware  to  modern  liquid 
crystal  displays,  but  in  many  areas  it  retained  the  legacy  layouts  of  information 
(Marchant,  Eastin,  &  Ferguson,  2001).  While  this  approach  solved  the 
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maintenance  and  supportability  issues,  it  did  not  resolve  any  of  the  legacy  human 
factors  concerns,  and  did  not  fully  leverage  the  advances  that  had  been  made  in 
glass  cockpit  technology.  The  system  still  displayed  graphics  and  text  in 
monochrome,  still  required  multiple  key-presses  to  read  system  and  subsystem 
information,  and  still  presented  information  in  closely  spaced  rows  of  digital 
numbers,  making  it  difficult  to  locate  off-normal  values  (McCandless  et  al. ,  2005). 


Figure  1 .  Shuttle  Glass  Cockpit  (From  NASA,  2000) 

To  address  these  remaining  shortcomings,  the  Cockpit  Avionics  Upgrade 
(CAU)  project  was  then  initiated  in  1999  (Marchant  et  al.,  2001).  CAU  was 
intended  to  update  the  display  formats  shown  on  the  cockpit  displays  in  order  to 
reduce  mental  workload  (MW),  enhance  situational  awareness  (SA),  and 
increase  performance,  thereby  improving  overall  system  safety  (McCandless, 
2004).  Based  on  human  factors  research,  the  program  produced  a  design  for 
enhanced  display  formats  and  keysets.  The  new  formats  were  intended  to  be 
task-oriented  and  tailored  to  each  phase  of  flight  (McCandless,  2004). 

Despite  demonstrating  considerable  improvement  over  the  baseline 
MEDS  cockpit  in  workload,  SA,  and  performance,  CAU  was  cancelled  in  2004, 
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and  no  orbiter  vehicle  ever  flew  with  the  enhanced  cockpit  (McCandless,  2004). 
To  evaluate  the  effectiveness  of  the  CAU,  thousands  of  data  points  on  human 
performance  were  collected  as  part  of  this  project,  but  since  the  cancellation, 
those  data  have  never  been  fully  analyzed  (McCandless,  2004). 

B.  PURPOSE 

As  part  of  the  CAU  evaluation,  the  data  were  analyzed  only  so  far  as  to 
determine  that  significant  improvement  did  exist  in  the  CAU  cockpit  in  MW  and 
SA.  To  date  a  complete  analysis  has  not  been  completed  to  assess  why 
astronaut  performance  improved,  what  design  changes  contributed  to  these 
improvements,  or  what  crew  activities  were  most  affected.  Furthermore,  the 
breadth  of  data  collected  allows  the  opportunity  to  examine  which  measures  are 
most  useful  in  predicting  overall  crew  performance.  Therefore,  there  are  three 
primary  objectives  of  this  thesis:  1)  complete  a  thorough  assessment  of  the  CAU 
evaluation  data  in  order  to  gain  understanding  of  why  an  improvement  exists,  2) 
develop  design  recommendations  for  future  manned  space  systems,  and  3) 
develop  recommendations  for  methods  of  evaluating  astronaut  performance  in 
these  future  systems. 

C.  PROBLEM  STATEMENT 

The  CAU  project  was  intended  to  leverage  over  20  years  of  advances  that 
had  been  made  in  human  factors  since  the  Shuttle  was  originally  designed  in  the 
1970s.  Despite  demonstrating  considerable  improvements  in  MW  and  SA,  CAU 
was  cancelled  in  2004  and  the  data  that  were  collected  during  the  program  was 
never  been  fully  analyzed.  With  the  retirement  of  the  Space  Shuttle,  the  impetus 
to  revisit  the  data  has  waned  as  well. 

It  is  inevitable  that  the  United  States  eventually  will  return  to  manned 
spaceflight  in  an  American  spacecraft.  When  this  happens,  it  is  essential  that 
existing  lessons  learned  in  safety,  situational  awareness,  and  mental  workload  all 
be  leveraged.  Through  the  CAU  program,  a  great  deal  of  data  have  already 
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been  collected  in  all  of  these  areas.  It  is  imperative  that  this  data  be  translated 
into  actionable  knowledge  that  can  be  applied  to  future  manned  spacecraft. 

In  examining  the  data  for  the  CAU  evaluation  study  the  following  research 

questions  are  proposed: 

1)  Did  the  CAU  design  changes  lead  to  improvements  in  crew 
mental  workload  and/or  situation  awareness? 

2)  If  mental  workload  and/or  situation  awareness  improvements 
exist,  do  they  vary  by  crew  station  or  phase  of  flight? 

3)  Can  crew  mental  workload  and/or  situation  awareness  changes 
be  traced  back  to  specific  CAU  design  modifications? 

D.  HUMAN  SYSTEM  INTEGRATION  DOMAIN  APPLICABILITY 

NASA  describes  Human  System  Integration  as  “an  umbrella  term  for 
several  areas  of  ‘human  factors’  research  that  include  human  performance, 
technology  design,  and  human-computer  interaction”  (NASA,  2012).  The 
Department  of  Defense  (DOD)  further  identifies  seven  domains  of  Human 
Systems  Integration:  human  factors  engineering  (HFE),  personnel,  habitability, 
manpower,  training,  safety  and  occupational  health,  and  survivability  (DOD, 
2008).  This  thesis  relates  to  HFE,  training,  and  system  safety  domains. 

HFE  is  defined  as  “the  comprehensive  integration  of  human 
characteristics  into  system  definition,  design,  development,  and  evaluation  to 
optimize  the  performance  of  human-machine  combinations”  (Booher,  2003). 
This  study  involves  the  application  of  HFE  principles  in  evaluating  a  set  of  display 
designs  of  the  Space  Shuttle.  The  lessons  learned  from  this  evaluation  can  be 
applied  to  future  activities  in  the  HFE  domain. 

Training  is  defined  as  the  “the  requisite  knowledge,  skills,  and  abilities 
needed  by  the  available  personnel  to  operate  and  maintain  systems  under 
operational  conditions”  (Booher,  2003).  The  design  modifications  evaluated  in 
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this  study  alter  the  task  requirements,  and  therefore  affect  the  knowledge  and 
skills  required  to  operate  the  system.  Consequently,  the  training  domain  is 
affected. 

System  Safety  is  defined  as  “the  inherent  ability  of  the  system  to  be  used, 
operated,  and  maintained  without  accidental  injury  to  personnel”  (Booher,  2003). 
The  primary  objective  of  the  CAU  program  was  to  improve  overall  system  safety 
by  improving  the  quality  of  the  human-computer  interface  (McCandless,  2004). 
Therefore,  this  evaluation  of  the  CAU  program  relates  to  the  system  safety 
domain. 

E.  ORGANIZATION 

This  thesis  is  organized  into  five  chapters.  Chapter  I  provides  an  overview 
of  the  Space  Shuttle  Cockpit  Avionics  Upgrade  program,  including  rationale  and 
objectives.  Chapter  II  provides  a  review  of  scientific  and  technical  literature 
regarding  Space  Shuttle  systems,  cockpit  evaluation,  and  flight  crew 
performance  assessment.  Chapter  III  describes  the  methods  by  which  the  CAU 
modifications  were  evaluated.  Chapter  IV  presents  the  results  of  the  evaluation. 
Chapter  V  offers  conclusions  and  recommendations  based  on  the  findings. 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  intent  of  this  chapter  is  to  provide  the  context  necessary  for 
understanding  and  evaluating  the  Shuttle  CAU  program.  It  begins  with  a 
historical  and  technical  review  of  shuttle  cockpit  avionics  systems.  Next  there  is 
a  review  of  relevant  methods  and  criteria  by  which  the  design  upgrades  can  be 
objectively  critiqued.  Finally,  there  is  a  review  of  the  theories  and  methods  by 
which  the  effectiveness  of  the  upgrades  can  be  assessed  experimentally.  The 
information  reviewed  includes  scientific  journals,  textbooks,  NASA  documents, 
technical  standards,  and  related  prior  work. 

The  literature  review  process  began  with  a  review  of  prior  work  in  the  CAU 
program.  This  included  materials  obtained  directly  from  the  NASA  Ames 
Research  Center,  as  well  as  those  that  were  found  by  searching  Google  Scholar 
for  the  terms  “Shuttle”  and  “CAU.”  To  provide  the  necessary  conceptual 
foundations,  the  search  was  extended  to  include  such  topics  as  mental  workload, 
situation  awareness,  and  information  processing.  Additional  electronic  resources 
searched  included  The  Naval  Postgraduate  School  Library,  Defense  Technical 
Information  Center,  and  NASA  Technical  Report  Server.  Available  textbooks 
were  also  reviewed  for  these  topics.  To  gain  a  practical  perspective  on  cockpit 
evaluation  methods,  additional  resources  were  obtained  from  the  U.S.  Naval  Test 
Pilot  School. 

B.  SHUTTLE  COCKPIT  SYSTEMS 

The  Space  Shuttle  was  one  of  the  first  major  applications  of  fly-by-wire 
control  technology  (Marchant  et  al.,  2001).  The  onboard  Data  Processing  System 
(DPS)  controls  almost  all  shuttle  operations  from  ascent  to  landing,  including 
flight  control,  system  management  (SM),  and  guidance  navigation  and  control 
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(GNC).  Most  tasks  during  ascent  and  entry  are  fully  automated,  with  the  crew 
monitoring  for  various  takeover  criteria  (NASA,  2002). 

At  the  heart  of  the  DPS  system  were  five  IBM  APC-101S  general  purpose 
computers  (GPCs)  (NASA,  2002).  Four  of  these  GPCs  ran  the  shuttle's  Primary 
Avionics  System  Software  (PASS),  while  the  fifth  ran  the  Backup  Flight  System 
(BFS).  PASS  was  designed  to  manage  all  shuttle  operations  from  launch  to 
landing.  BFS  was  much  simpler,  intended  to  provide  only  the  capabilities  needed 
to  safely  launch  and  recover  the  shuttle.  Each  software  suite  was  developed 
separately  to  minimize  the  possibility  that  a  programming  defect  in  one  would 
affect  the  other  (NASA,  2002).  Both  software  suites  performed  critical  functions, 
and  crewmembers  often  were  required  to  utilize  both  of  them  throughout  normal 
flight  operations  (Holland  &  Vanderark,  1993;  NASA,  2002). 

DPS  software  was  divided  by  phase  of  flight  into  Operational  Sequences 
(OPS),  and  further  subdivided  into  major  modes  (see  Figure  2).  Transitions 
between  OPS  were  initiated  by  the  crew,  but  transitions  between  MMs  could  be 
done  automatically  (NASA,  2002).  Each  MM  included  an  associated  display 
page  that  presented  the  crew  with  information  concerning  that  portion  of  the 
mission  phase  (NASA,  2002). 
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Figure  2.  Shuttle  Flight  Computer  Software  (NASA,  2002) 

In  order  to  discuss  the  shuttle's  glass  cockpit  in  context,  it  is  helpful  to 
begin  by  reviewing  the  layout  of  the  original  electro-mechanical  cockpit  (see 
Figure  3).  In  this  legacy  arrangement,  electrically  driven  mechanical  displays 
were  provided  for  monitoring  flight  performance  and  critical  systems  (Tranthien, 
1995).  This  included  an  Attitude  Director  Indicator  (ADI),  Horizontal  Situation 
Indicator  (HSI),  Alpha/Mach  Indicator  (AMI),  Altitude/Vertical  Velocity  Indicator 
(AVVI),  and  various  moving-tape  instruments  for  monitoring  critical  systems  such 
as  Main  Propulsion  (MPS),  Orbital  Maneuvering  (OMS),  and  the  Auxiliary  Power 
Unit  driven  Hydraulic  system  (APU/HYD).  Three  multi-function  cathode  ray  tube 
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(CRT)  displays  and  two  corresponding  keypads  were  provided  to  interface  with 
the  DPS.  Display  pages  unique  to  each  MM  were  presented  on  these  three 
displays.  Due  to  the  degree  of  computer  control  aboard  the  shuttle,  the  bulk  of 
operating  parameters  were  monitored  by  the  crew  solely  through  this  DPS 
interface  (Marchant  et  al.,  2001).  Each  CRT  Display/  Keyboard  pair  could  be 
switched  to  connect  to  any  of  the  five  GPCs  (NASA,  2002). 


ADI 


HSI 


Figure  3.  Legacy  Electromechanical  Cockpit  Layout  (From  Tranthien,  1995) 


The  original  (MEDS)  glass  cockpit  upgrade  had  nine  identical  Multifunction 
Display  Units  (MDUs)  installed  in  the  forward  panel  of  the  shuttle  cockpit  (see 
Figure  4)  (NASA,  2002).  Two  MDUs  were  installed  directly  in  front  of  the 
commander  and  pilot  stations,  respectively  labeled  CDR1,  CDR2,  PLT1,  and 
PLT2.  The  remaining  five  MDUs,  labeled  MFD1 ,  MFD2,  CRT  1 ,  CRT2,  and 
CRT3  are  located  between  the  two  crew  stations  (NASA,  2002). 
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Figure  4.  MEDS  Cockpit  Display  Layout  (From  NASA,  2002) 

The  nine  displays  are  driven  by  four  onboard  Integrated  Display 
Processors  (IDPs),  which  receive  information  from  five  data  busses  and  four 
analog-to-digital  converters  (NASA,  2002).  This  information  is  processed  for 
display  and  sent  to  the  MDUs  via  a  1553B  data  bus  (see  Figure  5).  The  CDR, 
PLT,  and  MFD  displays  are  each  connected  to  a  primary  and  secondary  IDP. 
The  CRT  displays  are  each  only  connected  to  a  single  IDP.  The  system  is 
configured  to  ensure  that  at  least  one  CDR,  one  PLT,  and  one  CRT  display 
remain  available  in  the  event  of  a  dual  IDP  failure.  The  five  GPCs,  along  with  the 
PASS  and  BFS  software,  were  not  affected  by  the  glass  cockpit  installation 
(Tranthien,  1995;  NASA,  2002). 
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Figure  5.  MEDS  System  Overview  (From  NASA,  2002) 
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The  Shuttle  Crew  Operating  Manual  (SCOM)  divides  the  set  of  available 
display  formats  in  the  baseline  MEDS  configuration  into  flight,  subsystem,  and 
DPS  formats  (NASA,  2002).  Flight  formats  display  graphical  representations  of 
the  legacy  electromechanical  flight  instruments  (ADI,  HSI,  AMI,  AVVI,  etc.). 
Subsystem  displays  include  APU/HYD  and  MPS/OMS  subsystem  vertical  tapes, 
as  well  as  the  Surface  Position  Indicator  (SPI).  The  DPS  display  formats  (which 
varied  according  to  the  current  MM)  were  carried  over  directly  from  the  legacy 
CRT  display  formats.  In  general,  the  CDR  and  PLT  MDUs  are  used  to  show  the 
flight  instruments,  the  MFD  MDUs  are  used  to  show  the  subsystem  displays  and 
the  CRT  MDUs  are  used  to  show  DPS  displays  (see  Figure  6). 


CRT  1 
DPS 


CRT  2 

DPS 


CDR  1 

CDR  2 

HSI/AMI 

ADI/AWI 

CRT  3 

DPS 


PLT  1 

PLT  2 
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OMS/MPS 

MFD  1 

OMS/MPS 


MFD  2 

HYD/APU 


Figure  6.  Typical  Baseline  Ascent  Display  Configuration  (From  NASA,  2002) 


The  CAU  called  for  minimal  hardware  upgrades,  however  the  display 
formats,  however,  differed  greatly  from  the  baseline  MEDS  (Hayashi  et  al., 
2005).  CAU  formats  were  divided  into  Flight,  System,  and  Fault  categories 
(Patrick  &  Mastracchio,  2002).  The  Flight  category  included  primary  flight  display 
(PFD),  trajectory  (TRAJ),  and  horizontal  situation  (H-SIT)  formats,  each  tailored 
to  the  current  MM.  The  System  category  included  specific  pages  for  ECLSS, 
DPS,  Navigation,  Control,  RCS,  OMS/MPS,  APU/Hyd,  and  EPS  subsystems. 
Fault  pages  provided  overall  failure  annunciation  and  diagnostic  information,  and 
were  intended  to  be  the  focal  point  for  critical  systems  monitoring  (Reisman, 
2002).  Figure  7  presents  a  typical  arrangement  of  CAU  display  formats. 
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Figure  7.  Typical  CAU  Display  Configuration  for  Ascent  (From  Reisman,  2002) 

C.  DESIGN  CRITERIA 

The  U.S.  Naval  Test  Pilot  School  Flight  Test  Manual  provides  a 
methodology  and  recommended  guidelines  for  conducting  a  cockpit  evaluation. 
A  central  element  of  this  evaluation  method  is  a  focus  on  the  mission  and  tasks 
to  be  performed  by  the  flight  crew  (Masters  et  al. ,  2005).  The  list  of  cockpit 
elements  to  be  evaluated  should  include  the  layout  of  displays  and  controls  as 
seen  from  the  design  eye  position,  the  hardware  attributes  of  the  various 
components,  the  formatting  of  display  content,  and  the  interrelationships  between 
controls  and  displays  (Masters  et  al.,  2005).  To  enable  a  thorough  cockpit 
evaluation  using  this  method,  it  is  helpful  to  review  the  relevant  aspects  of  task 
analysis,  cockpit  layout,  display  formatting,  hardware  attributes,  and  the  display- 
control  relationship. 

1.  Task  Analysis 

Task  analysis  is  defined  as  the  study  of  the  actions  and  cognitive 
processes  required  of  an  operator  to  achieve  a  system  goal  (Kirwan  &  Ainsworth, 
1992).  There  are  two  complementary  task  analysis  procedures  that  will  be 
employed  in  this  evaluation.  The  first  is  hierarchical  task  analysis  (HTA),  which  is 
a  method  used  to  define  an  activity  by  breaking  it  down  into  its  respective 
components  (Kirwan  &  Ainsworth,  1992).  The  second  method,  task 
decomposition,  is  a  way  of  expanding  the  information  from  the  task  description 
into  a  series  of  detailed  statements  which  are  of  analytical  interest  (Kirwan  & 
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Ainsworth,  1992).  For  this  evaluation,  the  goal  of  the  HTA  was  to  identify  the 
display-related  crewmember  tasks  during  ascent  and  entry  operations.  The  goal 
of  the  task  decomposition  was  to  identify  information  requirements,  challenges, 
and  opportunities  for  error  for  each  identified  task. 

Kirwan  and  Ainsworth  (1992)  identified  several  potential  means  of 
gathering  information  for  a  task  analysis,  including  observation,  interviews  with 
subject  matter  experts,  and  examination  of  documentation.  Potential  document 
sources  can  include  operating  manuals  and  emergency  procedures  (Kirwan  & 
Ainsworth,  1992).  Several  sources  are  available  for  gathering  information  on 
Space  Shuttle  flight  crew  tasks.  Most  notably,  the  SCOM  provides  an  overview 
of  all  normal  procedures  for  all  phases  of  flight,  and  the  crew  pocket  checklist 
provides  details  on  contingency  and  emergency  procedures  (NASA,  2002). 
Additionally,  Holland  and  Vanderark  (1993)  used  operating  manuals,  astronaut 
interviews,  and  observations  in  a  Space  Shuttle  simulator  to  conduct  a  detailed 
task  analysis  of  flight  crew  activities  during  entry  and  landing  phases  of  flight. 

2.  Design  Eye  and  Display  Placement 

The  Design  Eye  Position  is  defined  in  MIL-STD-1333B  as  the  intended 
viewpoint  of  the  average  operator  (see  Figure  8).  It  is  a  reference  point  from 
which  all  crew  station  dimensions  are  related  and  referenced  (DoD,  1987).  The 
placement  of  displays  within  a  cockpit  is  determined  with  this  point  in  mind.  For 
military  systems,  this  placement  is  standardized  in  MIL-STD-1333B  and  MIL- 
STD-1472G.  NASA-STD-3001  serves  a  similar  function  for  manned  spacecraft. 
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Figure  8.  Design  Eye  Position  (From  Woodson,  1981) 

The  NASA  standard  provides  three  relevant  guidelines  for  the  placement 
of  displays:  (1)  The  most  important  and  most  frequently  used  displays  shall  be 
provided  privileged  positions  in  the  crew's  viewing  zone  in  order  to  ensure  quick 
processing  and  reaction;  (2)  Displays  and  controls  shall  be  visible  and  be  within 
the  functional  reach  envelope  of  the  crew  under  all  conditions  (e.g.,  suited, 
seated,  restrained,  and  unrestrained);  and  (3)  Displays  and  controls  shall  be 
located  and  designed  so  that  they  may  be  used  to  the  required  degree  of 
accuracy  by  the  crew  in  normal  operating  positions  (NASA,  2011). 

Huchingson  (1981)  asserted  that  the  most  important  and  frequently  used 
instruments  should  be  grouped  together  within  a  30-degree  cone  of  vision 
centered  about  the  primary  line  of  sight.  Huchingson  further  identified  a 
maximum  visual  field  for  display  placement  as  +/-  35  degrees  horizontally,  and 
+40/-20  degrees  vertically  about  the  primary  line  of  sight.  These  limits  are  based 
on  eye  movement  alone,  without  head  movement  (Huchingson,  1981). 
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Figure  9.  Optimum  and  Maximum  Visual  Fields  (From  Huchingson,  1981) 

3.  Display  Design 

In  order  to  describe  a  cockpit  design  change  from  a  human  factors 
perspective,  it  is  helpful  to  first  identify  a  universal  framework  that  can  be  used  to 
evaluate  the  quality  of  a  cockpit  display.  One  such  widely  accepted  framework  is 
the  thirteen  principles  of  display  design  (Wickens,  Lee,  Liu,  &  Gordon-Becker, 
2004).  This  list  is  organized  into  four  distinct  categories:  (1)  perceptual 
principles,  (2)  mental  model  principles,  (3)  principles  based  on  attention,  and  (4) 
memory  principles. 


a.  Perception 

Perceptual  principles  influence  the  way  an  operator  initially  perceives  the 
information  being  displayed.  These  principles  emphasize  presenting  information 
clearly  and  unambiguously  so  as  to  promote  timely  perception  and  avoid 
confusion  (Wickens  et  al.,  2004).  Perception  can  be  further  aided  by  creating 
context,  or  by  using  familiar  representations,  such  as  fonts  and  icons  (Wickens  & 
Carswell,  2006).  Considerations  for  these  principles  are  summarized  in  Table  1 . 
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Principle 

Considerations 

Make  displays 
legible 

Consider  contrast,  visual  angle,  illumination, 
noise,  etc. 

Avoid  absolute 
judgment  limits 

Avoid  requiring  a  user  to  judge  more  than  5-7 
levels  of  a  single  variable  (color,  size,  etc). 

Top-down 

processing 

People  interpret  signals  based  on  what  they 
expect  to  perceive,  and  therefore  off-normal 
conditions  should  be  emphasized. 

Redundancy  gain 

Correct  interpretation  is  more  likely  when  a 
signal  is  expressed  in  more  than  one  place, 
especially  when  an  alternate  form  is  used. 

Use  discriminable 
elements 

Two  signals  that  look  alike  are  likely  to  be 
confused. 

Table  1 .  Perceptual  Principles  (From  Wickens  et  al. ,  2004) 


b.  Mental  Models 

Mental  model  principles  relate  to  the  operator's  expectations,  or  "mental 
model"  of  the  system  being  displayed  (Wickens  et  al.,  2004).  Mental  models  can 
be  defined  as  a  mechanism  to  generate  descriptions  of  system  purpose  and 
form,  explanations  of  system  functioning,  and  predictions  of  future  system  states 
(Rouse  &  Morris,  1985).  When  a  display  is  perceived,  operators  tend  to  interpret 
its  appearance  and  movement  in  terms  of  this  mental  model  (Wilson  & 
Rutherford,  1989).  Mental  Model  principles  are  summarized  in  Table  2. 


Principle  Considerations 

Principle  of  pictorial  The  display  should  look  like  the  variable  that  it 

realism  represents. 

Principle  of  the  Moving  elements  on  a  display  should  match  the 

moving  part  user's  mental  model  of  the  system  being 

represented. 


Table  2.  Mental  Model  Principles  (From  Wickens  et  al.,  2004) 
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c.  Attention 

Attentional  principles  are  important  for  displays  that  have  multiple 
elements  (Wickens  et  al 2004).  Three  components  of  attention  are  needed  to 
process  these  types  of  displays:  (1 )  selective  attention  (or  attention  allocation)  for 
choosing  the  necessary  information  source,  (2)  focused  attention  for  avoiding 
distraction  from  neighboring  sources,  and  (3)  divided  attention  for  allowing 
parallel  processing  from  two  sources  as  needed  (Wickens  et  al.,  2004).  Attention 
allocation  can  be  either  a  knowledge  drive  top-down  process,  or  a  bottom-up 
process  which  is  driven  by  the  salience  of  the  cue  (Yantis,  1993).  In  either  case 
it  can  be  inhibited  when  excessive  effort  is  required  to  shift  attention(Wickens  et 
al.,  2004).  The  attentional  principles  summarized  in  Table  3  provide  ways  to 
capitalize  on  human  strengths  or  mitigate  human  weaknesses  with  respect  to 
attention. 


Principle 

Considerations 

Minimizing 
information  access 
cost 

The  amount  of  effort  needed  to  move  between 
required  pieces  of  information  (menus  on  a 
display  screen,  checklist  pages,  etc)  should  be 
minimized. 

Proximity 

compatibility 

principle 

Two  related  pieces  of  information  that  must  be 
integrated  (such  as  a  graph  and  its  legend) 
should  be  displayed  in  close  proximity  to  each 
other.  Close  proximity  need  not  be  exclusively 
in  terms  of  space.  It  can  also  be  achieved  by 
using  a  common  color  or  pattern. 

Principle  of  multiple 
resources 

Divide  information  across  resources,  such  as 
visual  and  auditory,  to  facilitate  concurrent 
processing. 

Table  3.  Principles  Based  on  Attention  (From  Wickens  et  al.,  2004) 
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d.  Memory 


Memory  principles  deal  with  the  limited  capacities  of  both  long  term  and 
short  term  memory  (Wickens  et  al.,  2004).  Short  term  memory  is  heavily  limited 
in  the  number  of  “chunks”  of  information  that  can  be  retained  at  any  one  time 
(Cowan,  2010).  Long  term  memory  limitations  include  both  forgetting  important 
information,  as  well  as  persisting  in  following  outdated  or  incorrect  information 
(Wickens  et  al.,  2004).  These  principles,  presented  in  Table  4,  emphasize 
overcoming  these  limitations. 


Principle 

Considerations 

Replace  memory 
with  visual 
information 

Do  not  require  that  all  important  information  be 
retained  solely  in  working  memory  or  retrieved 
from  long  term  memory. 

Principle  of 
predictive  aiding 

Predicting  future  states  is  a  cognitively 
demanding  task. 

Principle  of 
consistency 

Displays  should  be  designed  in  a  way  that  is 
consistent  with  what  the  user  is  already  familiar 
with. 

Table  4.  Memory  Principles  (From  Wickens  et  al.,  2004) 


4.  Hardware  Attributes 

The  NASA  Space  Flight  Human-System  Standard  for  Human  Factors, 
Habitability,  and  Environmental  Health  (NASA-STD-3001  Volume  2)  defines  a  set 
of  technical  requirements  by  which  a  spacecraft’s  display  hardware  can  be 
evaluated  (see  Table  5).  Despite  the  fact  that  the  Space  Shuttle  is  not  a  military 
system,  MIL  STD  1472G  can  be  used  for  evaluation  as  well.  MIL-STD-1472  is 
widely  regarded  as  an  authoritative  source  of  good  human  factors  design 
practices  (Woodson,  1981). 
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Metric 

Minimum 

Maximum 

Notes 

Ambient  contrast  ratio 

Viewing  angle 

10 

-45  deg 

+45  deg 

Includes  ambient 
illumination 

4-point  viewing  angle 

Spatial  resolution 

Frame  rate 

Number  of  colors 

32  pixels/deg 

60  Hz 

4096 

(left,  right,  up,  down), 
contrast  and  color 
gamut  criteria  met 

Table  5.  Required  Visual  Display  Parameters  (From  NASA,  201 1 ) 


While  evaluating  against  the  NASA  standard  provides  a  sense  of  the 
overall  quality  of  a  display,  an  additional  level  of  analysis  is  needed  to  determine 
whether  or  not  existing  hardware  is  suitable  for  displaying  the  proposed  display 
formats.  To  this  end,  NASA  conducted  a  study  to  evaluate  the  color  and 
luminance  properties  of  the  existing  MDU  hardware  (McCandless,  2003).  Using 
a  colorimeter  and  a  luminance  meter,  measurements  were  taken  of  each 
required  color  from  crew  design  eye  positions  under  various  lighting  conditions. 
Two  relevant  key  findings  came  from  this  study:  1)  Viewing  angle  has  negligible 
effect  on  all  proposed  CAU  colors  except  orange,  which  can  vary  in  appearance 
from  yellow  to  red  depending  on  angle;  and  2)  The  display  can  not  produce  true 
black,  but  instead  appears  dark  blue  when  black  is  commanded. 

5.  Display  Control 

Fitts  and  Seeger  (1953)  identified  the  importance  of  the  relationship 
between  a  control  and  the  display  for  the  entity  being  controlled  in  the  principle  of 
stimulus-response  compatibility.  Good  compatibility,  which  is  associated  with 
timelier  and/or  more  accurate  response  to  stimuli,  is  achieved  when  the  control  is 
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located  in  close  proximity  to  the  display  of  the  entity  being  controlled  (Wickens  et 
al. ,  2004).  Stimulus-response  compatibility  is  also  affected  by  the  coding  (i.e. , 
verbal  vs.  spatial)  of  the  control  and  the  associated  display  (Wickens,  Vidulich,  & 
Sandry-Garza,  1984).  Design  requirements  based  on  these  principles  are 
articulated  in  NASA  STD  3001;  which  states  that  displays  and  controls  shall  be 
grouped  according  to  purpose  or  function,  and  that  the  relationship  between  the 
display  and  control  shall  be  logical  and  explicit  (NASA,  2011). 

MIL  STD  1472  provides  additional  guidelines  for  the  physical  properties  of 
display  control  hardware.  The  standard  specifies  that  a  physical  keyboard  shall 
be  used  for  entering  any  alpha-numeric  data,  and  that  positive  feedback  shall  be 
provided  on  screen  during  such  data-entry  tasks  (DoD,  2012).  It  further  requires 
that  individual  keys  in  a  vehicle-mounted  keypad  should  be  no  smaller  than  0.75 
inches  in  width,  and  be  separated  by  0.5  inches  (DoD,  2012).  Finally,  the 
standard  requires  that  all  keys  and  pushbuttons  (regardless  of  location)  shall 
provide  tactile  feedback  (DoD,  2012). 

D.  ASSESSING  DESIGN  EFFECTIVENESS 

MW  and  SA  are  both  considered  to  be  important  factors  in  the  design  of 
aviation  systems.  (Selcon,  Taylor,  &  Koritsas,  1991).  Tsang  and  Vidulich  (2006) 
noted  that  these  two  concepts  are  highly  interrelated,  and  are  affected  by  many 
of  the  same  factors.  Cockpit  design  improvement  efforts  are  often  focused  on 
these  common  factors,  with  an  overall  objective  of  reducing  mental  workload  and 
improving  SA  (e.g.,  Carmody-Bubb  &  Maybury,  1998;  Weinstein  &  Wickens, 
1992) 

Two  goals  of  the  CAU  program  were  to  reduce  workload  and  enhance 
situation  awareness.  Therefore,  prior  to  assessing  the  effectiveness  of  the 
program,  it  is  necessary  to  review  these  two  concepts.  Both  MW  and  SA  are 
reviewed  in  terms  of  underlying  theory,  measurement  technique,  and  practical 
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application.  Measurements  are  assessed  in  terms  of  validity,  sensitivity, 
selectivity,  diagnosticity,  intrusiveness,  and  reliability  (see  Table  6). 


Validity 

The  extent  to  which  a  dependent  measure  actually  assesses 

what  it  is  intended  to  measure 

Sensitivity 

the  degree  to  which  a  measure  distinguishes  between 

differing  conditions  or  states 

Selectivity 

the  degree  to  which  a  measure  is  sensitive  only  to  changes  in 

the  construct  of  interest 

Diagnosticity 

the  degree  to  which  a  measure  not  only  identifies  changes  but 

identifies  the  cause  of  any  variation 

Intrusiveness 

the  degree  to  which  a  measure  interferes  with  the  primary  task 

Reliability 

the  degree  to  which  a  measure  is  consistent 

Table  6.  Measurement  Criteria  (After  Marchant  et  al. ,  2001 ; 

Uhlarik  &  Comerford,  2002) 


1.  Mental  Workload 

a.  Theory  and  Issues 

The  most  basic  way  to  express  the  MW  concept  is  in  terms  of  spare 
capacity.  The  simplest  and  most  intuitive  way  to  express  spare  capacity  is  in 
terms  of  the  ratio  of  time  required  to  complete  a  set  of  tasks  and  the  time 
available  (TR/TA)  (Wickens  et  al.,  2004).  While  this  single  dimension  can  not 
fully  capture  MW’s  complexity  and  multidimensionality,  it  has  nonetheless  been 
shown  to  be  an  effective  approximation  in  some  situations  (Hendy,  Liao,  & 
Milgram,  1997).  In  addition  to  roughly  predicting  MW,  the  TR/TA  ratio  also 
should  predict  the  point  at  which  excess  workload  begins  to  degrade 
performance.  Whenever  time  required  exceeds  time  available  (TR/TA  >  1.0)  a 
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person  can  be  said  to  be  in  overload.  Conversely,  when  time  required  is  less 
than  time  available  (TR/TA  <  1 .0)  a  person  can  be  said  to  have  spare  capacity 
(Wickens  et  al. ,  2004).  In  situations  where  spare  capacity  exists,  the  amount  of 
spare  capacity  can  be  used  as  a  MW  measure  (Lysaght  et  al.,  1989;  Roscoe  & 
Ellis,  1990). 

While  time  makes  for  a  good  approximation  in  many  situations,  it  is  very 
limited  as  an  expression  of  overall  workload.  For  example,  tasks  that  are  very 
practiced  can  often  be  carried  out  with  little  conscious  thought,  regardless  of  how 
much  time  is  required  to  actually  complete  them  (Logan,  1985).  These  kinds  of 
issues  can  be  accounted  for  by  viewing  workload  not  in  terms  of  limitations  in 
time,  but  in  terms  of  a  limited  capacity  central  processor  (Moray,  1967).  This  view 
is  foundational  to  so-called  'capacity'  or  'resource'  theories,  which  postulate  that 
limited  attentional  resources  are  available  for  conducting  tasks,  and  that  more 
difficult  tasks  leave  fewer  resources  available  for  conducting  concurrent  tasks. 
The  available  resource  supply  fluctuates  based  on  the  individual's  level  of 
arousal,  and  performance  degradations  occur  when  resource  demands  exceed 
resource  supply  (Kahneman,  1973). 

As  a  result  of  studies  conducted  in  the  1970s,  evidence  emerged  that  task 

performance  could  not  be  attributed  to  a  single  resource  pool,  and  that  the 

presence  of  multiple,  separate  limited  resource  pools  appeared  to  better  account 

for  performance  (Wickens,  2002a).  The  presence  of  multiple  pools  related  to 

specific  abilities  is  a  foundational  principle  of  multiple-resource  theory.  Wickens's 

multiple-resource  model  proposes  four  dimensions  to  account  for  variance  in 

multitasking  performance:  (1)  processing  stages,  (2)  perceptual  modalities,  (3) 

visual  channels,  and  (4)  processing  codes.  In  each  dimension,  two  separate 

pools  of  resources  are  postulated  to  exist.  The  dimension  of  stages  refers  to 

information  processing  stages,  including  perception,  cognition,  and  response. 

Wickens  (2002)  identified  that  the  resources  used  for  perception  and  cognition 

appear  to  be  separate  and  distinct  from  those  resources  used  for  responding. 

The  dimension  of  modalities  refers  to  the  distinction  between  auditory  and  visual 
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modalities.  Wickens  identified  that  the  resources  used  for  each  of  these  two 
modalities  appears  to  be  separate  and  distinct.  The  dimension  of  channels 
refers  to  a  distinction  between  focal  and  ambient  visual  processing,  that  appear 
to  use  separate  resources.  Finally,  the  dimension  of  codes  refers  to  the 
distinction  between  spatial  and  verbal  processing.  Wickens  found  that  in  any 
cognitive  stage,  spatial  and  verbal  processes  appear  to  depend  on  separate 
resources.  Across  each  of  these  dimensions,  much  work  has  been  done 
suggesting  that  when  controls  and  displays  do  not  all  require  the  same  resource 
pool,  there  is  less  competition  for  limited  resources,  and  less  overall  workload  as 
a  result  (Lysaght  et  al .,  1989). 

b.  Measurement 

The  techniques  commonly  used  to  empirically  assess  MW  include  task 
performance  measures,  physiological  measures,  and  subjective  measures 
(Lysaght  et  al.,  1989).  Muckier  &  Seven  (1992)  noted  that  the  “distinction 
between  ‘objective’  and  ‘subjective’  measurement  is  neither  meaningful  nor 
useful  in  human  performance  studies.”  Casali  and  Wierwille  (1984)  found  that 
with  highly  trained  participants  (such  as  aircraft  pilots)  subjective  rating  scales 
are  highly  sensitive  instruments  for  measuring  MW.  Therefore,  of  the  three 
methods  listed  above,  subjective  measures  are  of  greatest  interest  to  this  study. 

Roscoe  and  Ellis  (1990)  showed  that  pilots  easily  adapt  to  subjectively 

expressing  MW  through  the  concept  of  spare  capacity,  and  developed  the 

Bedford  Workload  Rating  Scale  (BWRS)  based  on  this  concept.  The  BWRS, 

which  was  adapted  from  the  Cooper  Harper  rating  scale  for  aircraft  handling, 

uses  a  simple  decision  tree  (see  Figure  10)  to  arrive  at  a  subjective  assessment 

of  workload  on  a  scale  from  1  (Workload  insufficient)  to  10  (Pilot  unable  to  apply 

sufficient  effort).  While  easy  to  use,  the  authors  recognized  that  there  are 

significant  drawbacks  to  this  method.  One  of  the  most  glaring  is  the  fact  that  it 

requires  active  subject  participation,  and  therefore  cannot  be  used  to  measure 

workload  during  a  task  without  a  significant  disruption.  Another  important 

limitation  is  that  normalization  is  required  as  BWRS  results  tend  to  be  highly 
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individualized  to  each  pilot.  The  BWRS  has  been  extensively  used  in  cockpit 
workload  evaluation  studies,  and  is  well  understood  by  flight  crews  (Lysaght  et 
al. ,  1989).  It  has  been  found  to  correlate  well  with  other  MW  measures,  such  as 
heart  rate,  and  is  most  effective  when  used  during  or  shortly  after  the  flight  task 
in  question  (Roscoe  &  Ellis,  1990).  The  BWRS  is  also  somewhat  limited  in  its 
complexity  and  sensitivity. 


Figure  10.  Bedford  Workload  Rating  Scale  (From  Roscoe  &  Ellis,  1990) 


Because  workload  is  a  multidimensional  concept,  various  researchers 

have  argued  that  subjective  ratings  should  consist  of  more  than  just  a  single 

scale  (Wickens  et  al.,  2004).  In  developing  the  NASA  Task  Load  Index  (TLX), 

Hart  &  Staveland  (1988)  proposed  using  six  dimensions  to  assess  workload: 

mental  demand,  physical  demand,  temporal  demand,  performance,  effort,  and 

frustration.  Subjective  ratings  are  then  obtained  separately  for  each  of  these 

dimensions.  Each  dimension  is  weighted  according  to  how  much  it  contributed  to 

overall  workload,  and  an  aggregate  score  is  computed  by  summing  the  weighted 
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ratings  from  each  scale.  While  TLX  provides  more  dimensionality  than  BWRS,  it 
suffers  from  some  of  the  same  limitations.  It  cannot  be  used  to  measure 
workload  in  the  middle  of  a  task  without  disrupting  that  task,  and  it  requires 
normalization  for  each  participant.  Hart  (2006),  in  reviewing  the  use  of  TLX  over 
a  20  year  period  found  that  it  had  been  referenced  in  over  500  studies,  primarily 
in  aviation  settings.  A  literature  review  conducted  by  Cain  (2007)  indicated  that 
TLX  performs  comparably  to  other  multidimensional  workload  instruments,  and  is 
much  more  sensitive  than  one-dimensional  scales  such  as  the  BWRS.  A  similar 
review  conducted  by  Lysaght  (Lysaght  et  al.,  1989)  found  TLX  to  be  a  valid, 
reliable,  and  sensitive  technique  for  workload  assessment. 


1.  Mental  demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking, 

deciding,  calculating,  remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

I _ 1 _ I _ I _ I _ I _ 1 _ I _ I _ I _ 1 

Low  H  igh 

2.  P  hysical  demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling, 

turning,  controlling  activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk, 
slack  or  strenuous,  restful  or  laborious? 

1 _ I _ I _ I _ | _ I _ 1 _ I _ I _ I _ 1 

Low  H  igh 

3.  T  e  mporal  demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which 

the  tasks  or  task  elements  occurred?  Was  the  pace  s  low  and  leisurely  or  rapid  and 
frantic? 

I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ 1 

Low  H  igh 

4.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the 

task  set  by  the  researchers  (or  yourself)?  How  satisfied  were  you  with  your  performance 
in  accomplishing  these  goals? 

1 _ I _ I _ I _ I _ I _ I _ I _ I _ I _ 1 

Good  P  o  or 

5.  E  f  fort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your 

level  of  performance? 

I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ 1 

Low  H  igh 

6.  F  rustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed,  and  complacent  did  you  feel  during  the  task? 

I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ 1 

Low  H  igh 


Figure  1 1 .  NASA  TLX  Components  (From  Beutter  et  al.,  2006) 
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2. 


Situation  Awareness 


a.  Theory  and  Issues 

Endsley  (1999)  defined  SA  as  "the  perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the  comprehension  of  their 
meaning  and  the  projection  of  their  status  in  the  near  future."  This  is  the  most 
frequently  cited  definition  to  be  found  in  the  scientific  literature  (Uhlarik  & 
Comerford,  2002).  Encapsulated  within  Endsley’s  definition  are  three  distinct 
levels  of  SA.  Level  1  SA  involves  perceiving  critical  elements  in  the  environment. 
Level  2  SA  involves  understanding  these  critical  elements  and  relating  them  to 
the  aircrew's  goals.  Level  3  SA  involves  projection  these  elements  into  the  near 
future  and  understanding  what  will  happen.  Wickens  (2002b)  further  identified  a 
clear  distinction  between  spatial  awareness  and  system  awareness,  with  distinct 
requirements  for  each  type.  Spatial  awareness  requires  knowledge  of  elements 
such  as  attitude,  heading,  velocity,  vertical  velocity,  aircraft  capabilities,  and 
projected  flight  path;  whereas  System  awareness  requires  awareness  of  such 
elements  as  system  status,  settings,  and  the  impact  of  system  configurations  and 
malfunctions  on  overall  system  performance  and  flight  safety  (Wickens,  2002b) 

As  a  process,  SA  is  closely  linked  with  perception  and  working  memory, 
and  is  therefore  subject  to  limits  in  attentional  capacity  (Tsang  &  Vidulich,  2006). 
Consequently,  high  workload  tasks  that  consume  attentional  resources  can  lead 
to  poor  SA.  Therefore,  the  way  in  which  a  cockpit  presents  information  to  the 
operator  has  a  huge  impact  on  crew  SA  (Tsang  &  Vidulich,  2006).  Presenting 
too  much  information  at  one  time  can  lead  to  excessive  workload,  which  can  be 
as  much  of  a  hindrance  to  good  SA  as  presenting  too  little  information  (Uhlarik  & 
Comerford,  2002).  In  general,  Endsley  (1999)  notes,  "The  more  complex  the 
systems  are  to  operate,  the  greater  the  increase  the  mental  workload  required  to 
achieve  a  given  level  of  SA.  When  that  demand  exceeds  human  capabilities,  SA 
will  suffer." 
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b.  Measurement 

In  their  review  of  the  scientific  literature  pertaining  to  MW  measurement, 
Uhlarik  &  Comerford  (2002)  identified  three  categories  of  SA  measurement: 
Explicit  measures,  implicit  measures,  and  subjective  measures.  Explicit 
measures  are  those  that  simply  ask  the  subject  to  report  information  about  the 
recent  state  of  an  aircraft;  implicit  measures  are  those  that  infer  a  level  of  SA 
from  other  measures  of  task  performance;  and  subjective  measures  are  based 
solely  on  opinion. 

Explicit  measures  can  provide  specific  detailed  information  about  the 
operator's  concept  of  a  situation  that  can  then  be  objectively  evaluated  against 
reality  (Endsley,  1995).  This  method  typically  takes  the  form  of  questionnaires 
that  can  be  administered  following  the  completion  of  a  task,  during  the  course  of 
its  execution,  or  while  the  task  is  paused  (DoD,  2012;  Uhlarik  &  Comerford,  2002; 
Wickens  et  al. ,  1984;  etc.).  Each  approach  has  its  benefits  and  drawbacks. 
Post-task  questionnaires  allow  the  subject  ample  time  to  respond  to  lengthy 
questionnaires,  but  are  limited  by  the  level  of  detail  that  can  be  remembered 
throughout  an  entire  task  (Endsley,  1995).  Questions  asked  during  the 
execution  of  a  task  overcome  the  memory  limitation,  but  may  impose  additional 
workload  that  can  alter  performance.  The  intrusiveness  of  the  measure  can  be 
reduced  when  the  task  can  be  frozen,  as  by  pausing  a  simulator  and  blanking  the 
displays  (Tsang  &  Vidulich,  2006). 

One  such  tool  that  employs  a  “freeze  technique”  is  the  Situation 
Awareness  Global  Assessment  Tool  (SAGAT)  (Endsley,  1995).  This  tool  is 
based  on  developing  a  global  list  of  SA  requirements  for  a  given  system  and 
task,  and  then  asking  the  subject  questions  based  on  this  list  at  random  intervals 
throughout  the  simulation.  SAGAT  is  generally  considered  to  offer  high  sensitivity 
to  changes  in  situation  awareness,  but  it  can  be  intrusive  when  the  simulator  is 
resumed  (Tsang  &  Vidulich,  2006). 


28 


Implicit  measures  are  objective,  non-intrusive,  and  relatively  easy  to 
collect  in  a  simulator,  but  they  are  limited  in  their  selectivity  and  sensitivity 
(Endsley,  1995).  These  measures  can  be  collected  globally,  or  for  just  a  specific 
sub-task  of  interest.  Global  measures  are  not  particularly  reliable  for  measuring 
SA  due  to  the  multitude  of  cognitive  factors  (many  unrelated  to  SA)  that  can 
affect  overall  performance  (Uhlarik  &  Comerford,  2002).  Imbedded  task 
measures  offer  improved  selectivity  over  global  measures,  however  the  results 
may  be  misleading  because  changes  in  a  single  component  of  SA  can  greatly 
impact  other  components  of  SA  (Endsley,  1995).  For  this  reason,  imbedded 
task  measures  are  rarely  used  (Uhlarik  &  Comerford,  2002). 

Subjective  measures  of  SA  are  inexpensive  and  easy  to  use,  but  have 
many  significant  limitations.  Self-rated  subjective  measures  of  SA  are  inherently 
limited  by  the  fact  that  an  operator  (by  definition)  does  not  know  whether  his  own 
SA  is  incomplete  or  inaccurate  (Uhlarik  &  Comerford,  2002).  Furthermore, 
subjective  self-ratings  tend  to  be  confounded  by  the  effects  of  performance  and 
workload  (Endsley,  1995).  Conversely,  observer-rated  measures  of  SA  are 
inherently  limited  in  what  the  observer  can  know  about  an  operator's  concept  of  a 
situation.  An  observer  can  reliably  detect  overt  errors  related  to  SA,  but  little 
more  (Uhlarik  &  Comerford,  2002). 

3.  Using  Multiple  Measures 

There  is  no  consensus  in  the  literature  on  a  single  preferred  instrument  for 
measuring  mental  workload  or  situation  awareness.  Selcon,  Taylor,  and  Koritsas 
(1991)  acknowledged  that  measures  of  workload  and  SA  have  a  lot  in  common, 
but  each  captures  certain  aspects  of  performance  that  the  other  one  does  not. 
Rubio,  Diaz,  Martin  and  Puente  (2004)  evaluated  several  such  measures  of 
workload  for  intrusiveness,  sensitivity,  validity,  diagnosticity,  and  acceptance,  and 
found  that  no  single  measure  outperformed  the  others  in  every  category.  Tsang 
and  Vidulich  (2006)  noted  several  situations  that  could  lead  to  dissociation  of 

measures  of  SA  and  workload,  a  condition  they  say  can  be  informative,  “if  one  is 
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cognizant  of  the  idiosyncratic  properties  of  the  different  measures.”  As  a  result  of 
this  diversity  of  opinion,  multiple  measures  of  workload  and  SA  are  often  used  in 
aviation  human  performance  studies.  The  choice  of  which  measures  to  use  is 
situation  dependent,  and  requires  the  experimenter  to  have  good  working 
knowledge  of  the  strengths  and  limitations  of  each  of  the  available  tools  (Lysaght 
etal.,  1989;  Muckier  &  Seven,  1992). 

E.  SUMMARY 

The  design  of  the  Space  Shuttle’s  data  processing  system  provides  the 
flight  crew  with  a  large  volume  of  information  gathered  from  a  variety  of  onboard 
and  external  sources.  The  chief  objective  of  this  literature  review  has  been  to 
survey  the  human  factors  principles  that  relate  to  the  way  in  which  this 
information  is  organized  and  presented  in  a  cockpit  design.  Together,  these 
principles  support  a  method  of  critically  evaluating  a  cockpit  design  change  and 
assessing  its  effectiveness. 

The  critical  evaluation  must  begin  with  a  thorough  task  analysis  in  order  to 
gain  an  understanding  of  what  is  required  of  the  design.  With  this  in  mind,  the 
evaluation  must  consider  the  layout  of  the  displays  from  the  operator  perspective, 
the  formatting  of  information  on  each  display,  the  hardware  attributes,  and  the 
display-control  relationship.  Guidelines  have  been  established  for  each  of  these 
characteristics. 

In  order  to  assess  the  effectiveness  of  design  changes,  measures  of  flight 
crew  mental  workload  and  situation  awareness  are  needed.  Several  such 
measures  can  be  found  in  the  literature,  each  with  its  own  advantages  and 
disadvantages.  Multiple  measures  are  therefore  needed  to  ensure  a  thorough 
assessment.  Careful  analysis  of  MW  and  SA  data  can  show  what  improvements 
exist  and  where.  Tracing  these  results  back  to  the  critical  design  evaluation  can 
offer  insight  into  why.  The  use  of  multiple  measures  of  MW  and  SA  presents  an 
opportunity  to  examine  which  measures  work  best  in  this  setting. 
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III.  METHODS 


A.  OVERVIEW 

The  evaluation  begins  with  a  critique  of  the  CAU  program  design 
modifications,  and  then  proceeds  to  an  assessment  of  the  effectiveness  of  these 
changes.  The  design  critique  is  centered  around  a  task  analysis  of  crew 
operations  in  both  ascent  and  entry  phases  of  flight.  It  examines  the 
modifications  in  terms  of  cockpit  layout,  design  principles,  hardware  attributes, 
and  display  control.  Paired  astronaut  crews  flying  identical  simulated  scenarios  in 
the  baseline  and  upgraded  CAU  configurations  were  then  assessed  using  MW 
and  SA  measures.  These  results  were  then  analyzed  for  statistical  significance. 

B.  DESIGN  CRITIQUE 

The  design  changes  between  the  baseline  and  CAU  cockpit  display 
formats  were  evaluated  according  to  established  human  factors  principles. 
Central  to  this  evaluation  was  a  task  analysis  of  flight  crew  activities  during 
ascent  and  entry  phases  of  flight.  This  task  analysis  was  used  to  define  a  set  of 
information  display  requirements.  From  this,  the  cockpit  display  upgrades  were 
evaluated  in  terms  of  cockpit  layout,  display  design,  hardware  attributes,  and 
display  control.  The  evaluations  were  conducted  using  a  wide  variety  of 
documents  that  describe  Space  Shuttle  cockpit  hardware  and  software 
characteristics  (e.g.,  Jenkins,  2010;  Thomsen  &  Hancock,  1994;  Tranthien, 
1995). 


1.  Task  Analysis 

Two  complementary  task  analysis  techniques  were  used  to  evaluate  the 
design  changes.  First,  a  basic  hierarchical  task  analysis  was  used  to  identify  the 
display-related  crewmember  tasks  during  ascent  and  entry  operations.  Once 
display-related  tasks  were  identified,  the  HTA  was  not  developed  further.  Based 

on  this  HTA,  a  task  decomposition  was  then  used  to  identify  information 
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requirements  and  display  support  in  each  cockpit  for  each  identified  task.  The 
results  of  this  stage  of  the  evaluation  were  used  to  guide  subsequent  stages. 

2.  Design  Eye  and  Display  Placement 

The  astronauts’  field  of  view  must  be  taken  into  consideration  for  this 
critique.  Although  no  changes  were  made  in  the  physical  arrangement  of  display 
hardware,  changes  made  to  the  display  formats  can  affect  the  availability  of  key 
pieces  of  information.  This  portion  of  the  critique  therefore  began  by  finding  the 
design  eye  position  for  each  crew  station.  Next,  optimum  and  maximum  visual 
fields  were  used  to  determine  what  displays  are  most  readily  visible  to  each  crew 
station.  The  optimum  and  maximum  fields  were  based  on  the  limits  established 
by  Huchingson  (1981),  defined  in  the  previous  chapter.  Viewing  angle  limits  of 
the  display  hardware  were  considered  as  well.  Finally,  by  using  typical  layouts  of 
display  formats  for  both  the  baseline  and  CAU  cockpits,  lists  were  developed  for 
each  crew  station  detailing  what  information  is  most  readily  visible. 

3.  Display  Formats 

Every  display  format  in  the  shuttle  cockpit  was  examined  and  modified  as 
part  of  the  CAU  program.  In  addition,  the  organizational  structure  was  modified 
such  that  there  is  not  a  one-to-one  mapping  of  old  display  formats  to  new  ones. 
Display  formats  were  variously  added,  deleted,  consolidated,  and  rearranged  in 
order  to  meet  the  design  objectives. 

For  this  critique,  the  display  formats  were  grouped  according  to  task. 
Task  categories  include:  (1)  Monitoring  flight  parameters  (altitude,  speed, 
heading,  etc.);  (2)  Systems  Management;  and  (3)  Monitoring  trajectory.  Display 
formats  within  each  of  these  categories  share  certain  characteristics  that  allowed 
them  to  be  evaluated  collectively.  Within  each  category,  differences  were 
highlighted  between  the  CAU  and  baseline  displays.  These  differences  are  then 
critiqued  according  to  the  thirteen  principles  of  display  design. 
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4.  Display  Hardware 

Although  no  changes  have  been  made  to  the  shuttle’s  MDU  hardware, 
there  are  nonetheless  some  hardware  attributes  worth  examining  in  this 
evaluation.  The  display  hardware  was  examined  first  for  its  general  suitability 
within  the  cockpit,  and  then  for  its  suitability  in  displaying  the  CAU  display 
formats.  The  first  part  of  the  critique  was  conducted  by  checklist  method,  wherein 
display  hardware  specifications  are  evaluated  against  NASA  and  DoD  standards. 
The  second  part  of  the  critique  was  done  by  comparing  MDU  color  capabilities 
with  the  requirements  of  the  CAU  formats. 

5.  Display  Control 

Although  no  major  changes  were  made  to  the  layout  of  the  keypads  and 
bezel  keys  used  to  control  the  displays,  changes  to  the  display  formats  can 
significantly  affect  the  display-control  relationship.  This  portion  of  the  critique 
examined  that  relationship.  The  hardware  characteristics  of  the  controls  were 
first  evaluated  according  to  NASA  and  DoD  standards  to  examine  their  suitability 
for  data  entry  and  display  configuration  tasks.  Next,  the  relationships  between 
the  control  hardware  and  the  information  displays  were  evaluated  for  both  the 
baseline  and  CAU  designs.  The  evaluation  focused  on  principles  of  stimulus- 
response  compatibility,  and  examined  both  proximity  and  coding. 

C.  DESIGN  ASSESSMENT 

The  testing  process  was  designed  around  a  set  of  minimum  success 
criteria,  which  provided  the  standard  by  which  the  updated  displays  would  be 
evaluated  (McCandless,  2004).  Achieving  the  minimum  success  criteria  would 
indicate  that  the  program  was  successful,  whereas  failure  to  meet  the  criteria 
would  have  resulted  in  either  a  redesign  of  the  displays,  or  cancellation  of  the 
entire  project.  The  goals  for  the  minimum  success  criteria  were  baselined  by  the 
Office  of  Spaceflight  at  NASA  Headquarters  and  confirmed  during  four 


33 


independent  reviews  (McCandless,  2004).  The  minimum  success  criteria  were 
divided  into  three  categories,  presented  in  Table  7. 


Category 

Definition 

Goals 

Situation 

Awareness 

Situation  awareness  is 
defined  as  a  crew  member's 
understanding  of  his  or  her 
dynamic  flight  environment 
with  respect  to  the  mission. 

a)  Trajectory,  horizontal  situation, 
alternate  landing  sites,  abort 
determination  and  monitoring: 

100%  improvement 

b)  DPS,  EPS,  MPS,  OMS 
monitoring  and  failure  recognition: 
50%  improvement 

c)  Other  system  monitoring  and 
failure  recognition:  20% 
improvement 

Mental 

Workload 

Workload  is  defined  as  the 
mental  effort  necessary  to 
perform  a  task. 

Reduce  workload,  as  measured  by 
Bedford  and  NASA  TLX 

Performance 

Performance  is  a  measure  of 
how  well  the  crew 
accomplished  the  appropriate 
tasks  in  the  cockpit. 

a)  Reduction  in  unidentified 
malfunctions  and  recognition  time 

b)  Reduction  in  errors 

c)  Reduction  in  keystrokes 

Table  7.  Minimum  Success  Criteria  (After  McCandless,  2004) 


1.  Participants 

The  evaluation  was  conducted  with  six  crews  of  three  astronauts  each. 
Each  crew  included  a  flown  pilot  or  commander  in  the  commander  seat,  an 
unflown  pilot  in  the  pilot  seat,  and  an  unflown  mission  specialist  in  the  mission 
specialist  2  (MS2)  seat.  These  three  positions  are  the  primary  "stake-holders"  in 
the  CAU  upgrade,  which  makes  them  the  appropriate  subjects  for  the  evaluation. 
The  goal  with  this  crew  make-up  was  to  test  the  displays  with  relatively  new 
astronauts  who  would  have  the  least  amount  of  ingrained  experience  and 
repetition  with  the  Baseline  displays,  yet  who  also  had  some  experience  with 
shuttle  operations. 
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Training  in  the  baseline  configuration  was  not  required  because  of  the 
level  of  familiarity  the  participants  had  already  acquired  through  several  years  of 
experience  and  training.  Training  in  the  CAU  configuration  was  conducted  over 
the  course  of  3-4  weeks.  It  consisted  of  20  hours  of  classroom  training,  5  hours 
of  Dynamic  Skills  Trainer  (DST)  training,  and  10  hours  of  familiarization  sessions 
in  the  Shuttle  Mission  Simulator. 

2.  Instruments 

a.  Subjective  Workload 

The  workload  measures  taken  for  the  evaluation,  the  BWRS  (see  Table  9) 
and  the  NASA  TLX  (see  Table  10),  are  subjective  assessments.  Both  methods 
were  discussed  in  the  previous  chapter.  Workload  measures  were  collected  from 
each  crewmember  at  the  completion  of  each  run. 


Rating  Description 

1  Workload  insignificant. 

2  Workload  low. 

3  Enough  spare  capacity  for  all  desirable  additional  tasks. 

4  Insufficient  spare  capacity  for  easy  attention  to  additional  tasks. 

5  Reduced  spare  capacity.  Additional  tasks  cannot  be  given  the  desired 

amount  of  attention 

6  Little  spare  capacity.  Level  of  effort  allows  little  attention  to  additional  tasks. 

7  Very  little  spare  capacity,  but  maintenance  of  effort  in  the  primary  tasks  is 

not  in  question 

8  Very  high  workload  with  almost  no  spare  capacity.  Difficulty  in  maintaining 
level  of  effort. 

9  Extremely  high  workload.  No  spare  capacity.  Serious  doubts  as  to  ability  to 
maintain  level  of  effort. 

10  Task  abandoned.  Unable  to  apply  sufficient  effort. 

Table  8.  Bedford  Workload  Rating  Scale  (After  McCandless,  2004) 
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Component 


1)  Mental  demand  (from  low  to  high  on  a  scale  of  1  to  10):  How  much  mental  and 
perceptual  activity  was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or  demanding,  simple 
or  complex,  exacting  or  forgiving? 

2)  Physical  demand  (from  low  to  high  on  a  scale  of  1  to  10):  How  much  physical 
activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling  activating,  etc.)? 
Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or 
laborious? 

3)  Temporal  demand  (from  low  to  high  on  a  scale  of  1  to  10):  How  much  time 
pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or  task  elements 
occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

4)  Performance  (from  good  to  poor  on  a  scale  of  1  to  10):  How  successful  do  you 
think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the  researchers  (or 
yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these 
goals? 

5)  Effort  (from  low  to  high  on  a  scale  of  1  to  10):  How  hard  did  you  have  to  work 
(mentally  and  physically)  to  accomplish  your  level  of  performance? 

6)  Frustration  (from  low  to  high  on  a  scale  of  1  to  10):  How  insecure,  discouraged, 
irritated,  stressed  and  annoyed  versus  secure,  gratified,  content,  relaxed,  and 
complacent  did  you  feel  during  the  task? 


Table  9.  NASA  TLX  Rating  Components  (After  McCandless,  2004) 


b.  Situation  Awareness 

The  SA  data  can  be  divided  into  objective  and  subjective  categories. 
Objective  questions  were  those  that  had  a  definitive  correct  or  incorrect  answer. 
These  questions  were  based  on  the  SAGAT  discussed  in  the  previous  chapter. 
Objective  SA  questions  related  to  trajectory  management,  critical  system  status, 
and  non-critical  system  status.  An  example  of  an  objective  question  is: 

What  was  your  trajectory  energy  state  at  the 
beginning  of  the  run  (circle  one)? 

High  Low  Nominal  Don't  Know 
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Subjective  ratings  were  based  on  crewmembers'  opinions,  and  had  no 
correct  or  incorrect  answer.  Subjective  SA  questions  related  to  trajectory 
management,  critical  system  status,  and  non-critical  system  status.  An  example 
of  a  subjective  question  is: 

Rate  your  situational  awareness  of  the  fuel  cell 
problem  as  provided  by  the  displays. 

123456789 10 

The  subjective  rating  scale  used  for  rating  situation  awareness  is 
presented  in  Table  8: 

Rating  Description 

Insufficient  SA 

1  Completely  unaware  of  situation 

2  Mostly  unaware  of  situation  and  totally  unable  to  monitor/follow-up 

3  Somewhat  aware  of  situation  and  mostly  unable  to  monitor/follow-up 
Reduced  SA 

4  Somewhat  aware  of  situation  and  somewhat  able  to  monitor/follow-up 

5  Mostly  aware  of  situation  and  somewhat  able  to  monitor/follow-up 
Adequate  SA 

6  Mostly  aware  of  situation  and  mostly  able  to  monitor/follow-up 

7  Completely  aware  of  situation  and  somewhat  able  to  monitor/follow-up 
Excellent  SA 

8  Completely  aware  of  situation  and  mostly  able  to  monitor/follow-up 

9  Completely  aware  of  situation  and  completely  able  to  monitor/follow-up 
Perfect  SA 

10  Completely  aware  of  situation,  completely  able  to  monitor/follow-up,  and 
aware  of  next  worst  failure 

Table  10.  Description  of  Situation  Awareness  Scale  (After  McCandless,  2004) 
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3. 


Observers 


One  to  two  observers  were  in  the  cockpit  for  every  run,  all  were  either 
astronauts  or  members  of  the  NASA  evaluation  team.  Additional  observers  were 
in  the  instructor  station.  Cockpit  observers,  as  trained  evaluators,  provided  a 
unique  perspective  on  non-verbal  communication.  These  observers  provided 
subjective  data  by  estimating  crew  situation  awareness  and  workload. 
Observers  in  the  instructor  station  were  used  to  monitor  crew  actions  and  record 
objective  data  on  errors,  unrecognized  malfunctions,  and  recognition  times. 

4.  Apparatus 

a.  Simulator 

The  evaluation  was  conducted  in  the  Shuttle  Mission  Simulator  (SMS)  at 
NASA  JSC  in  Houston,  Texas.  This  full  motion  simulator  replicates  all  aspects  of 
the  shuttle  cockpit  and  provides  high-fidelity  simulation  of  shuttle  ascent  and 
entry  operations.  For  this  evaluation,  the  simulator  was  configured  with  two 
classes  of  displays:  Baseline,  and  CAU.  Baseline  displays  are  the  Orbiter 
Increment-29  (01-29)  displays  in  the  MEDS  cockpit.  CAU  displays  are  the  01-41 
redesigned  displays. 

The  simulator  was  used  to  record  the  number  of  key  presses  on  the  DPS 
keypads.  These  data  were  divided  into  key  presses  related  to  display  navigation, 
and  those  use  for  other  functions.  The  number  of  display  edge  keys  pressed  and 
switches  thrown  was  also  recorded. 

b.  Flight  Data  File 

The  evaluation  only  included  ascent  and  entry  flight  phases.  The  five  FDF 
procedure  books  associated  with  ascent  and  entry  were  updated  to  account  for 
the  technical  changes  imposed  by  the  CAU  system.  Additional  FDF 
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modifications  were  avoided  to  eliminate  the  potential  of  misinterpreting  FDF 
improvements  as  cockpit  improvements.  The  Baseline  testing  evaluations  used 
the  FDF  from  shuttle  flight  STS-1 12. 

5.  Procedure 

The  evaluation  process  was  centered  around  the  execution  of  scripted 
formal  scenarios  simulated  with  both  the  baseline  and  CAU  cockpit  displays. 
These  scenarios  included  normal  ascents,  aborts  (including  RTLS,  TAL,  and 
ATO),  normal  entry  (under  both  PASS  and  BFS  guidance),  and  off  normal  entry. 

The  purpose  of  the  Baseline  testing  was  to  evaluate  the  existing  Baseline 
shuttle  cockpit  to  provide  a  basis  of  comparison  for  the  CAU  cockpit.  After  an 
interlude  of  8  to  11  months,  the  CAU  displays  were  evaluated  under  the  same 
conditions  as  the  Baseline  testing.  The  interlude  was  intended  to  reduce  the 
chances  that  the  crews  would  remember  the  details  of  the  scripted  scenarios 
during  the  testing  on  the  CAU  displays.  Baseline  testing  was  run  over  October  - 
December,  2003,  whereas  CAU  testing  was  run  over  August  -  September,  2004. 

For  each  cockpit,  the  crews  participated  in  three  data  collection  sessions, 
each  lasting  about  2  hours.  Each  session  included  eight  scripted  runs,  which 
were  identical  in  both  baseline  and  CAU  evaluation.  Session  1  was  comprised  of 
entry  runs  and  sessions  2  and  3  were  comprised  of  ascent  runs.  The  runs 
consisted  of  short  (approximately  10  minute)  time  slices  of  a  given  flight  phase. 
The  start  and  end  point  for  each  script  varied  based  on  the  required  test 
objectives.  Each  run  contained  several  simulated  malfunctions,  which  were 
designed  to  test  the  effect  of  the  modifications  to  the  displays.  Throughout  each 
scenario,  observers  recorded  various  objective  measures  of  crew  performance. 
At  the  end  of  each  script,  the  simulator  was  frozen  and  the  questionnaires  were 
administered  to  the  crew. 
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D.  DATA  COLLECTION  AND  ANALYSIS 

Table  1 1  provides  a  summary  of  the  data  collected  in  each  run.  Data  for 
each  measure  were  collected  in  both  the  ascent  and  entry  phases  of  operations, 
and  in  both  the  baseline  and  CAU  design  configurations.  Individual  measures 
were  collected  for  each  of  the  six  commanders,  pilots,  and  mission  specialists. 


Source 

Category 

Subcategory 

Measures 

Individual 

MW 

Subjective 

BWRS 

(by  crew 

NASA  TLX 

position) 

SA 

Subjective 

Trajectory  Awareness 

Critical  System  Awareness 
Non-critical  System  Awareness 

Objective 

Trajectory  Status  Questions 
Critical  System  Status 

Non-critical  System  Status 

Observer 

MW 

Objective 

BWRS 

SA 

Subjective 

Trajectory  Awareness 

Critical  System  Awareness 
Non-critical  System  Awareness 

Performance 

Objective 

Number  of  Errors 

Malfunction  Recognize  Time 

Simulator 

Performance 

Objective 

Navigation  Key  Presses 
Non-navigation  Key  Presses 
Switches  Thrown 

MDU  Edge  Key  Presses 

Table  1 1 .  List  of  MW,  SA,  &  Performance  Measures  Collected 


The  data  were  examined  with  a  series  of  statistical  tests  intended  to 
address  the  research  questions  of  this  thesis.  The  first  research  question  asked 
if  the  CAU  design  changes  led  to  improvements  in  MW  and  SA.  It  was  then 
asked  whether  or  not  these  changes  were  greater  for  one  specific  crew  position 
or  phase  of  operations  versus  the  others.  These  two  questions  required  a 
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statistical  evaluation  of  how  MW  and  SA  were  affected  by  differences  in  design, 
crew  position,  and  phase  of  operations.  To  this  end,  a  series  of  Analysis  of 
Variance  (ANOVA)  were  conducted. 

Because  of  data  set  limitations,  not  every  measure  was  included  in  the 
ANOVA.  Some  measures  were  under-sampled  for  certain  conditions,  and  some 
errors  were  made  in  recording  data.  Ultimately,  only  one  measure  of  MW,  and 
one  type  of  SA  measure  were  used.  The  decision  of  which  measures  to  use  was 
based  on  available  sample  sizes,  and  variance  within  each  measure.  For  MW, 
BWRS  results  were  used.  For  each  of  the  components  of  SA  (Trajectory,  Critical 
Systems,  and  Non-Critical  Systems),  subjective  measures  were  used. 

An  initial  round  of  ANOVA  tests  were  conducted  that  included  a  nested, 
within-subjects  design  to  examine  the  significance  of  individual  differences.  The 
results  of  these  tests  indicated  that  the  effect  of  subjects  was  insignificant. 
Having  demonstrated  that  individual  differences  are  minimal,  the  design  was 
reduced  to  a  purely  between-factors  ANOVA.  Although  this  approach  bends  the 
assumption  of  strict  independence  of  observations,  this  deviation  is  appropriate 
given  the  homogeneity  of  the  sample  population. 

Ultimately,  the  ANOVA  tests  for  MW  and  SA  included  three  fixed  factors: 
design  (Baseline  vs  CAU),  phase  of  operations  (Ascent  vs  Entry),  and  position 
(CDR  vs  PLT  vs  MS2).  For  each  of  these  tests,  results  were  considered 
statistically  significant  if  analysis  rejected  the  null  hypothesis  of  no  difference 
between  levels  of  a  factor  or  interaction  between  factors  at  an  alpha  level  of  .05. 
Where  significant  effects  were  found,  post-hoc  analysis  was  conducted  using 
Tukey’s  Honestly  Significant  Difference  (HSD)  test. 
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IV.  RESULTS 


A.  DESIGN  CRITIQUE 

The  organization  of  information  in  the  baseline  MEDS  cockpit  is 
constrained  by  the  inherent  limitations  of  the  legacy  cockpit  from  which  it  is 
functionally  derived.  Information  was  divided  according  to  its  source,  which  could 
include  analog  instruments,  PASS,  and  BFS  software.  As  a  result,  information 
about  one  system  or  even  one  flight  parameter  is  sometimes  scattered  across 
two  or  more  display  formats,  not  all  of  which  can  be  viewed  at  any  one  time. 
Extensive  display  navigation  is  sometimes  needed  to  gather  required  system  and 
subsystem  information.  Furthermore,  some  tasks  (such  as  identifying  available 
landing  sites  based  on  energy  state)  require  that  the  crew  consult  printed  Flight 
Data  Files  (FDF),  which  contain  the  tables  needed  to  manually  perform  the 
required  calculation. 

A  design  objective  of  the  CAU  program  was  to  resolve  these  problems  by 
consolidating  information  from  multiple  sources  onto  a  single  display  to  create 
display  formats  that  are  more  task-oriented.  These  formats  include  single-source 
system  summary  pages  (such  as  MPS  Sum,  APU/FIYD  Sum,  etc),  as  well  as 
single-source  pages  for  monitoring  trajectory  performance  (such  as  Ascent  TRAJ 
and  Entry  TRAJ). 

1.  Task  Analysis 

The  results  of  the  hierarchical  task  analysis  for  ascent  and  entry  display- 
related  tasks  are  presented  in  Figures  12  and  13,  respectively.  In  general,  tasks 
can  be  divided  into  GNC  and  SM  functions  that  are  performed  concurrently 
throughout  Ascent  and  Entry  phases.  Each  phase  also  includes  an  additional  set 
of  tasks  associated  with  responding  to  off-nominal  conditions.  Each  bottom-level 
task  presented  represents  a  requirement  for  an  information  display.  These 
requirements  are  further  decomposed  in  the  sections  that  follow. 
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Figure  12.  Display  Related  Tasks  During  Ascent  and  Abort 


Figure  13.  Display  Related  Tasks  During  Entry 
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The  manner  in  which  these  information  requirements  are  met  differs 
greatly  between  the  baseline  and  CAU  cockpit  configurations.  These  differences 
were  examined  through  a  task  decomposition  process  in  which  specific 
information  requirements  for  each  task  were  identified,  along  with  the  associated 
displays  within  each  cockpit  design.  These  results  are  presented  in  Tables  12 
and  13  for  Ascent  and  Entry  operations,  respectively.  Of  note,  a  minimum  of  10 
necessary  display  formats  were  identified  in  the  baseline  design,  compared  to 
four  in  the  CAU. 


Operation 

Information 

Requirements 

Baseline  Formats 

CAU  Formats 

Monitor  Maneuvers 

Vehicle  Attitude 

ADI/AVVI  and  HSI/AMI; 
or  Comp  FI 

PFD 

Track  Trajectory 
(Ascent) 

DPS  Trajectory  data 

BFS  Ascent  TRAJ 
and/or  PASS  Ascent 
TRAJ 

Ascent  TRAJ 

Monitor  MPS 
Performance, 
Throttling 

Power  setting,  fuel 
remaining 

OMS/MPS  Sum 

Ascent  TRAJ 

Track  Abort 
Boundaries 

DPS  Trajectory  data 

BFS  Ascent  TRAJ 

Ascent  TRAJ 

Monitor  Navigation 
Performance 

A  between  PASS  and 
BFS  guidance 

BFS  Ascent  TRAJ  and 
PASS  Ascent  TRAJ 

Ascent  TRAJ 

Monitor  for  System 
Faults 

Failure  annunciation 

Fault  Sum 

Fault  Sum 

Diagnose  System 
Faults 

Parameters  for  MPS, 
OMS,  DPS,  EPS, 
ECLSS,  APU/Hyd, 

Nav,  RCS  systems 

BFS  Sys  Sum,  GNC 

Sys  Sum,  MPS/OMS 
sum,  APU/Hyd  Sum 

CAU  System-Specific 
Displays 

Abort  Guidance 

DPS  Trajectory  data, 
Primary  Flight 
Instruments 

ADI/AVVI,  HSI/AMI, 

BFS  TRAJ  (DPS),  and 
PASS  Ascent  TRAJ 

PFD, 

Ascent  TRAJ 

Determine 

Available  Landing 
sites 

Runways  reachable 
based  on  current 
energy  state 

Relayed  by  radio  or 
computed  by  energy 
state  and  cue  cards 

H  Sit 

Table  12.  Summary  of  Display  Related  Tasks  During  Ascent 
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Operation 

Information 

Requirements 

Baseline  Formats 

CAU  Formats 

Monitor  Maneuvers 

Vehicle  Attitude 

ADI/AVVI  &  HSI/AMI; 
or  Comp  FI 

PFD 

Track  Trajectory 
(Entry) 

DPS  Trajectory  data 

BFS  &/or  PASS  Entry 
Trajectory 

Entry  TRAJ 

Track  Energy  state  & 
available  landing  sites 

Runways  reachable 
based  on  current 
energy  state 

Relayed  by  radio  or 
computed  by  energy 
state  and  cue  cards 

H  Sit 

Monitor  Navigation 
Performance 

A  between  PASS  and 
BFS  guidance 

BFS  Entry  TRAJ  & 
PASS  Entry  TRAJ 

Entry  TRAJ 

Monitor  for  system 
faults 

Failure  annunciation 

Fault  Sum 

Fault  Sum 

Diagnose  system 
faults 

Parameters  for  MPS, 
OMS,  DPS,  EPS, 
ECLSS,  APU/Hyd, 

Nav,  RCS  systems 

BFS  Sys  Sum,  GNC 
Sys  Sum,  MPS/OMS 
sum,  APU/Hyd  Sum 

CAU  System-specific 
Displays 

Fly  Entry  manually 

Basic  flight 
instruments,  DPS 
computed  trajectory 

HSI/AMI  &  ADI/AVVI, 
BFS  or  PASS  Entry 
TRAJ 

PFD,  Entry  TRAJ 

Table  13.  Summary  of  Display  Related  Tasks  During  Entry 


Task  responsibilities  are  different  for  each  crew  position.  For  redundancy, 
individual  crew  positions  are  assigned  overlapping  Primary  and  Secondary 
responsibilities.  These  requirements  can  be  varied  by  individual  crews,  but  a 
typical  distribution  is  presented  in  Table  14. 
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Position 

Primary 

Secondary 

CDR 

Track  Trajectory 

Track  Abort  Boundaries 

Monitor  DPS,  ECLSS  systems 

Monitor  OMS,  RCS,  Flight  Control 
Systems 

PLT 

Monitor  MPS,  EPS,  APU/HYD,  OMS, 
RCS,  and  Flight  Control  Systems 

Track  Trajectory 

MS2 

Overhead  Panels 

Track  Abort  Boundaries 

Overhead  Gauges 

Consult  Reference  Data  (cue  cards) 

Table  14.  Typical  Flight  Crew  Duties  During  Ascent  and  Entry 

(From  NASA,  2002) 


2.  Design  Eye  and  Display  Placement 

No  changes  were  made  to  the  shuttle  seating  positions  or  display 
locations  as  a  result  of  the  CAU  program.  Therefore,  DEPs  and  fields  of  view  for 
both  the  baseline  and  the  upgraded  cockpits  are  identical.  The  DEPs  of  the 
Commander  and  Mission  Specialist  crew  stations  are  presented  from  front,  side, 
and  overhead  views  in  Figures  14,  15,  and  16,  respectively.  The  position  of  the 
pilot  station  mirrors  that  of  the  commander  station. 


Note:  Linear  units  are  inches  and  angular  units  are  degrees. 

Figure  14.  Side  View  of  Shuttle  Cockpit  Measurements 

(From  McCandless,  2003) 
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Note:  Linear  units  are  inches  and  angular  units  are  degrees. 

Figure  15.  Side  View  of  Shuttle  Cockpit  Measurements 

(From  McCandless,  2003) 


Note:  Linear  units  are  inches  and  angular  units  are  degrees. 

Figure  16.  Top  View  of  Shuttle  Cockpit  Measurements 

(From  McCandless,  2003) 
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The  optimum  visual  field  for  the  commander’s  station  was  determined  to 
be  a  16-inch  diameter  circle  centered  on  the  top  edge  of  the  CDR2  display  unit. 
This  circle  encompasses  the  entire  CDR2  display,  and  the  rightmost  portions  of 
the  CDR1  display.  The  maximum  field  of  view  was  determined  to  be  a  rectangle 
42  inches  wide,  and  35  inches  in  height.  This  rectangle  includes  the  remainder 
of  the  CDR1  display,  as  well  as  the  complete  CRT1  and  MFD  1  units.  Allowing 
for  head  movement,  the  commander  is  limited  only  by  the  viewing  angle  limits  of 
the  display  hardware.  Given  these  limits,  the  commander  can  see  all  MDUs  with 
the  exception  of  the  two  PLT  units.  The  pilot’s  side  mirrors  the  commander’s 
side.  The  visibility  of  each  MDU  from  the  Commander’s  position  is  summarized 
in  Figure  17. 


Within  hardware  viewing  angle  limits 
(Head  movement  required) 

Not  viewable 


Figure  17.  Display  Availability  for  Commander  Position 

The  optimum  visual  field  for  the  mission  specialist  station  was  determined 
to  be  a  32-inch  diameter  circle  centered  on  the  CRT3  display.  This  circle 
encompasses  all  five  of  the  central  MDUs.  The  maximum  field  of  view  was 
determined  to  include  the  entire  forward  instrument  panel.  The  Mission 
Specialist  is  within  hardware  viewing  angle  limits  for  all  MDUs,  however  the  view 
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of  the  outboard-most  units  is  partially  obstructed  by  the  commander  and  pilot 
seats.  The  visibility  of  each  MDU  from  the  Mission  Specialist’s  position  is 
summarized  in  Figure  18. 


Within  optimum  field  of  view 
Within  maximum  field  of  view 
Within  hardware  viewing  angle  limits 
(Head  movement  required) 

Not  viewable 


Figure  18.  Display  Availability  for  MS2  Position 


In  the  baseline  configuration,  the  arrangement  of  display  formats  across 
each  of  the  MDUs  can  be  varied  based  on  crew  preference  and  phase  of  flight. 
A  typical  display  layout  for  the  baseline  configuration  is  described  in  the  Shuttle 
Crew  Operations  Manual  (see  Figure  19).  Using  this  typical  configuration,  the 
relative  accessibility  of  each  display  format  was  determined  for  each  crew 
position  (see  Table  15). 


CRT  1 

CRT  2 

DPS 

DPS 

CDR1 

CDR  2 

CRT  3 

PLT  1 

PLT  2 

HSI/AMI 

ADI/AWI 

DPS 

COMP  F/l 

OMS/MPS 

MFD  1 

MFD  2 

OMS/MPS 

HYD/APU 

Figure  19.  Typical  Baseline  Display  Configuration 
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Position 

Displays  within 
optimum  FOV 

Displays  within 
maximum  FOV 

Displays  visible  with 
head  movement 

CDR 

ADI/AVVI 

HSI/AMI 

DPS  (closest) 
OMS/MPS 

HYD/APU 

DPS 

PLT 

COMP  FI 

OMS/MPS 

HYD/APU 

DPS  (closest) 

DPS 

MS2 

DPS 

OMS/MPS 

HYD/APU 

ADI/AVVI 

COMP  FI 

Table  15.  Display  Format  Availability  in  a  Typical  Baseline  Configuration 


As  with  the  baseline  design,  the  layout  of  display  formats  across  each  of 
the  MDUs  can  be  varied.  A  typical  display  layout  for  the  CAU  configuration  is 
described  in  the  CAU  display  dictionary  (see  Figure  20).  Using  this  typical 
configuration,  the  relative  accessibility  of  each  display  format  was  determined  for 
each  crew  position  (see  Table  16). 


HSIT 


QMS 


Fault 

Sum 


TRAJ 


PFD 


DPS 


Figure  20.  Typical  CAU  Display  Configuration 
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Position 

Displays  within 

Displays  within 

Displays  visible  with 

optimum  FOV 

maximum  FOV 

head  movement 

CDR 

PFD 

T  rajectory 

Fault  Summary 

DPS  Status 

MPS 

Horizontal  Situation 

OMS 

PLT 

T  rajectory 

PFD 

Fault  Summary 

MPS 

DPS  Status 

OMS 

Horizontal  Situation 

MS2 

Fault  Summary 

PFD 

DPS 

Horizontal  Situation 
MPS 

QMS 

T  rajectory 

Table  16.  Display  Format  Availability  in  a  Typical  CAU  Configuration. 


3.  Display  Formats 

Display  format  changes  were  evaluated  according  to  the  principles  of 
display  design  (see  Chapter  II).  Several  dozen  distinct  display  formats  exist  in 
each  cockpit  configuration,  and  there  is  no  direct  relationship  between  Baseline 
and  CAU  formats.  Therefore,  to  evaluate  design  changes,  displays  were 
grouped  according  to  function.  Comparisons  were  made  using  representative 
example  display  formats  from  each  cockpit  design.  Where  applicable,  design 
changes  were  described  in  terms  of  Perceptual  Principles,  Attentional  Principles, 
Mental  Model  Principles,  and  Memory  Principles. 

a.  Primary  Flight  Instruments 

Display  formats  were  provided  in  both  baseline  and  CAU  cockpits 
that  present  graphical  representations  of  legacy  ADI,  HSI,  AMI,  and  AVVI 
instruments.  These  instruments  provided  information  about  basic  flight 
parameters,  such  as  attitude,  speed,  altitude,  rate  of  climb,  angle  of  attack,  and 
heading.  In  the  baseline  cockpit,  this  information  was  provided  by  the  HSI/AMI 
(see  Figure  21)  and  ADI/A VVI  (see  Figure  22)  display  formats.  Alternately,  a 
composite  ADI/HSI  display  (see  Figure  23)  was  provided  that  combined  these 
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instruments  onto  a  single  page.  In  the  CAU  cockpit,  this  information  was 
provided  solely  by  the  Primary  Flight  Display  (PFD)  format  (see  Figure  24).  The 
arrangement  of  these  instruments  in  the  CAU’s  PFD  format  was  similar  to  the 
baseline  cockpit's  Composite  ADI/HSI  display,  but  some  changes  were  made  in 
the  presentation  of  the  information.  The  changes  are  discussed  in  the  sections 
that  follow. 


Figure  21.  Baseline  HSI/AMI  Display 


Figure  23.  Baseline  Composite  ADI/HSI  Display 


Figure  24.  CAU  Primary  Flight  Display  (Ascent  Configuration) 


Perceptual  and  Attentional  Principles 

The  changes  made  in  the  use  of  color  are  immediately  apparent  when 
comparing  the  two  displays.  While  the  baseline  display  used  several  inconsistent 
colors  to  display  its  various  elements,  the  CAU  display  presents  a  single,  unified 
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color  scheme.  In  the  baseline  display,  separator  lines  and  non-critical  elements 
sometimes  have  as  much  contrast  with  the  background  as  the  messages 
themselves.  For  example,  the  boxes  surrounding  the  numerical  readouts  are 
colored  in  green  and  magenta,  and  the  backgrounds  for  the  rate  sliders  around 
the  ADI  are  blue.  This  unnecessary  contrast  decreases  the  relative  salience  of 
the  important  signals,  thereby  hindering  perception.  The  CAU  cockpit  format 
presents  these  less  critical  elements  in  dark  grey  to  reduce  their  contrast  with  the 
background,  while  presenting  important  signals  in  white,  maximizing  contrast. 

A  similar  effect  was  achieved  by  minimizing  the  clutter  in  the  bottom  half  of 
the  display.  The  format  of  the  accelerometer  (g-meter),  for  example,  was 
changed  from  a  prominently  colored  moving  tape  display  in  the  baseline  cockpit 
to  a  less  salient  dial  and  pointer  display  in  the  CAU.  Other  display  elements 
have  simply  been  removed  when  not  needed.  This  reduction  in  clutter  makes  it 
easier  to  maintain  focused  attention  on  the  more  critical  elements. 

The  location  of  the  digital  readouts  associated  with  each  of  the  vertical 
tape  instruments  was  changed  from  being  above  the  tapes  in  the  baseline 
cockpit,  to  being  centered  on  the  tapes  in  the  CAU.  This  change  exploits  the 
proximity  compatibility  principle,  as  it  makes  it  easier  for  the  astronaut  to 
mentally  integrate  the  digital  readout  with  the  moving  tape.  In  addition,  this 
change  brings  the  digital  readouts  in  line  with  the  centerline  of  the  ADI,  enabling 
a  quick  scan  in  a  straight  horizontal  line.  This  helps  to  minimize  the  cost  of 
moving  selective  attention  from  one  instrument  to  the  next. 

Memory  Principles 

With  the  exception  of  the  previously  mentioned  accelerometer,  CAU 
instruments  move  and  operate  in  the  same  way  as  the  baseline  instruments. 
The  CAU  displays  therefore  remain  consistent  with  the  experience  of  the  existing 
user  base,  and  do  not  require  extensive  re-learning  to  understand. 


55 


b. 


Guidance  and  Navigation 


In  general,  there  are  two  types  of  display  formats  used  throughout 
both  cockpits  that  provide  guidance  and  navigation  information:  (1)  Trajectory, 
and  (2)  Horizontal  situation.  Trajectory  displays  (sometimes  called  vertical 
situation  displays)  present  a  side  view  of  the  shuttle’s  trajectory,  and  horizontal 
situation  displays  present  a  top-down  view.  These  formats  are  task  specific,  with 
different  variants  being  used  at  each  stage  of  ascent,  entry,  and  abort.  However, 
these  formats  had  many  common  characteristics  and  were  evaluated  collectively. 

In  the  baseline  cockpit,  this  information  was  provided  by  a  variety  of 
MM-specific  DPS  formats.  For  ascent,  only  trajectory  formats  were  provided 
(see  Figure  25).  For  entry,  both  trajectory  and  horizontal  situation  formats  were 
provided.  In  each  case,  separate  formats  were  provided  by  PASS  and  BFS 
software.  In  the  CAU  cockpit,  both  trajectory  and  horizontal  situation  formats 
were  provided  for  both  ascent  and  entry  (see  Figures  26  and  27).  Information 
from  PASS  and  BFS  software  was  integrated  into  a  single  page.  The  changes 
are  discussed  in  the  sections  that  follow. 


Figure  25.  Baseline  BFS  Ascent  Trajectory 

56 


v«j  n«  i 

103 

HET  0  ✓  da  0?  1 

•  Horn 

III 

pns  ea  30 

ie<  104  i04 

bf  aa  30 

T^o  Oh  14 

u  ad  75  I 

'  p  ee 

H  L  86 

• 

•  • 

Dump  TTC  101 

0 

Jnh  Incl 

A 

l  ■» Vi 

:>P  1  £0  Jnh 

Oty  Side  )5 

Figure  26.  CAU  Ascent  Trajectory  Display 

Perceptual  Principles 

The  baseline  formats  used  a  simple  green  on  black  color  scheme  for  all 
information.  The  CAU  formats,  in  comparison,  used  multiple  colors  to  distinguish 
between  various  classes  of  information  (see  Appendix  B).  This  change 
increases  the  salience  and  discriminability  of  the  important  signals,  thereby 
improving  perception. 

Attentional  Principles 

The  CAU  displays  consolidated  information  that,  in  the  baseline  cockpit, 
was  spread  across  multiple  displays.  For  example,  the  CAU  ascent  trajectory 
format  (see  Figure  25)  combines  the  relevant  portions  of  the  baseline  PASS  and 
BFS  ascent  trajectory  displays.  In  addition,  the  CAU  format  incorporates 
performance  information  for  the  main  propulsion  system,  which  must  be  mentally 
integrated  with  trajectory  information  during  ascent.  In  the  baseline  cockpit,  MPS 
performance  information  was  provided  on  a  separate  display  format.  This 
change  exploits  the  proximity  compatibility  principle,  and  reduces  the  division  of 
attention  needed  to  perform  trajectory  monitoring. 
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The  CAU's  ascent  horizontal  situation  display  (see  Figure  27)  has  no 
direct  antecedent  in  the  baseline  cockpit.  This  display  format  provides 
information  about  the  horizontal  flight  path  and  abort  options  during  ascent.  This 
information  was  not  provided  on  any  display  in  the  baseline  cockpit.  The 
determination  of  abort  options  was  computed  automatically  at  mission  control 
center,  and  was  relayed  to  the  crew  by  voice  communication  when  needed.  In 
the  event  of  communication  failure,  the  determination  had  to  be  done  manually 
by  reference  to  primary  flight  instruments  and  a  printed  flight  data  file. 
Incorporating  this  information  into  a  single  display  greatly  reduces  the  information 
access  cost  associated  with  understanding  abort  options. 
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Figure  27.  CAU  Ascent  Horizontal  Situation  Display 

c.  Systems  &  Fault  Management 

System  summary  display  formats  were  developed  for  the  CAU  in  order  to 
consolidate  status  information  about  ECLSS,  DPS,  Navigation,  Control,  RCS, 
QMS,  MPS,  APU/HYD,  and  EPS  subsystems.  While  each  system's  display  is 
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unique,  they  all  have  many  characteristics  in  common  and  were  therefore 
evaluated  collectively.  The  changes  are  discussed  in  the  sections  that  follow. 

Attentions I  Principles 

In  the  baseline  cockpit,  subsystem  information  was  spread  across  multiple 
displays,  organized  according  to  its  source.  For  example,  system  parameters 
that  were  monitored  by  BFS  software  were  found  on  the  BFS  system  summary 
display  format  (see  Figure  28).  Some  other  system  parameters  were  presented 
on  dedicated  display  formats  (e.g.,  MPS/OMS  summary,  see  Figure  29).  The 
CAU  display  formats  were  instead  organized  according  to  the  subsystem(s) 
being  represented.  Information  for  a  given  subsystem  was  pulled  from  all 
available  sources,  and  presented  on  a  single  display  (see  Figure  30).  The  CAU 
design  changes,  therefore,  both  exploited  the  proximity  compatibility  principle, 
and  reduced  the  effort  needed  to  shift  selective  attention  between  different 
information  sources. 
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Figure  29.  Baseline  OMS/MPS  Display 
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Perceptual  Principles 

In  the  baseline  cockpit,  most  system  information  was  presented  in  the 
form  of  green  text  over  a  black  background.  Numbers,  labels,  and  dividing  lines 
were  all  presented  in  the  same  color.  The  upgraded  CAU  formats  instead 
exploited  color  to  enable  the  crew  to  differentiate  between  the  various  classes  of 
information,  especially  during  off-nominal  conditions  (see  Appendix  B). 

Mental  Model  &  Memory  Principles 

Baseline  and  CAU  cockpits  differ  greatly  in  the  use  of  graphics.  While  the 
baseline  display  formats  used  matrices  of  numbers  to  convey  information  about  a 
system,  the  CAU  formats  (where  appropriate)  incorporated  this  information  into  a 
rudimentary  system  schematic.  The  schematics  depicted  critical  relationships 
between  various  system  components.  Icons  were  added  to  show  the  status  of 
valves,  and  malfunctions  are  indicated  by  changing  the  color  of  the  represented 
component.  Instead  of  using  a  matrix  of  numbers,  important  values  were  placed 
on  the  schematic  according  to  the  component  they  relate  to. 

4.  Display  Hardware 

No  changes  were  made  to  the  shuttle  display  hardware  as  a  result  of  the 
CAU  program.  Therefore,  both  the  baseline  and  the  upgraded  cockpits  have  the 
same  physical  attributes.  Each  MDU  consists  of  a  6.7  inch  square  full-color 
active  matrix  liquid  crystal  display  (AMLCD),  a  brightness  control  knob,  six 
programmable  edge  keys,  and  an  on-off  switch.  All  of  the  displays  are  readable 
in  sunlight  and  can  be  viewed  from  +/-  60  degrees  horizontally,  and  -10  deg/+45 
degrees  vertically.  They  have  a  pixel  density  of  172  ppi,  a  refresh  rate  of  60Hz, 
and  a  high  ambient  contrast  ratio  exceeding  6:1  (Thomsen  &  Hancock,  1994).  All 
of  these  parameters  are  within  current  NASA  standards  with  the  exception  of 
ambient  contrast  ratio,  which  is  required  to  by  10:1.  For  comparison,  current 
department  of  defense  standards  only  require  ambient  contrast  ratio  of  5:1  for 
this  type  of  display  (DoD,  2012). 
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5.  Display  Control 

In  both  the  baseline  and  CAU  configurations,  display  content  is  controlled 
through  display  edge-keys  and  keypads.  No  changes  were  made  to  the 
locations  of  these  keys.  The  edge  keys  are  used  to  navigate  among  available 
display  formats.  The  keypads  are  used  to  provide  data  entry  within  certain 
display  formats.  In  the  baseline  configuration,  this  data  entry  was  only  applicable 
to  the  three  CRT  display  units  when  displaying  DPS  formats.  In  the  CAU 
configuration,  data  can  be  entered  into  any  display  unit  whenever  an  appropriate 
format  is  being  displayed  (McCandless,  2004).  This  change  allows  for  a  greater 
level  of  proximity  compatibility  when  data  entry  is  done  on  displays  that  are 
closer  to  the  keypad  (such  as  the  MFD  or  CDR2  display),  but  it  also  opens  the 
possibility  for  poor  proximity  compatibility  when  data  entry  is  done  on  more  distal 
displays. 

NASA  STD  3001  requires  that  there  be  a  “logical  and  explicit”  relationship 
between  a  control  and  its  associated  display.  For  display  navigation,  this 
requirement  is  met  equally  with  the  display  edge  keys  in  both  the  CAU  and 
baseline  cockpit.  For  data  entry,  this  requirement  is  met  by  both  the  baseline 
and  CAU  configurations,  but  is  best  achieved  in  the  CAU  configuration  when 
displays  formats  requiring  data  entry  are  selected  on  the  MDUs  that  are  closest 
to  the  keypad. 

B.  DESIGN  ASSESSMENT 

1.  Mental  Workload 

The  results  of  the  ANOVA  for  MW  are  presented  in  Table  17.  The  three- 
way  interaction  was  not  found  to  be  significant.  Significant  two-way  interaction 
effects  were  found  between  Design  and  Phase  of  Operations,  and  between 
Design  and  Position.  Both  of  these  effects  were  found  to  be  significant  at  the 
p<. 01  level.  The  results  also  revealed  a  highly  significant  main  effect  for  Design. 
This  effect  was  far  greater  than  that  of  any  interaction,  and  is  therefore  worth 
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addressing.  Plots  of  the  data,  Figures  31  and  32,  reveal  that  the  CAU  design 
greatly  outperforms  the  Baseline  design  in  terms  of  MW. 


Source  of  Variation 

Sum  of  Squares 

df 

Mean  Square 

F 

P 

Design 

1503.34 

1 

1503.34 

643.79 

0.000 

Ops 

49.80 

1 

49.80 

21.30 

0.000 

Position 

8.40 

2 

4.20 

1.80 

0.267 

Design  x  Ops 

23.04 

1 

23.04 

9.86 

0.002 

Design  x  Position 

29.54 

2 

14.77 

5.05 

0.002 

Ops  x  Position 

12.76 

2 

6.38 

2.73 

0.066 

Design  x  Ops  x  Position 

10.54 

2 

5.27 

2.25 

0.106 

Error 

1655.50 

708 

2.34 

Total 

3448.99 

719 

Table  17.  ANOVA  for  Mental  Workload 


To  identify  the  details  of  Design/Operations  interaction,  the  differences 
between  Baseline  and  CAU  designs  were  plotted  for  both  Ascent  and  Entry 
operations  (see  Figure  31).  Between  Baseline  and  CAU  designs,  crews 
appeared  to  show  a  greater  decrease  in  MW  during  ascent  phases  when 
compared  to  entry  phases. 

Post  hoc  comparisons  were  used  to  verify  the  significance  of  this 
interaction.  The  mean  workload  measure  for  the  Baseline  Ascent  condition 
(M=7.0)  was  found  to  be  significantly  different  from  the  Baseline  Entry  condition 
(M=6.1).  However,  the  CAU  ascent  condition  (M=3.7)  did  not  significantly  differ 
from  the  CAU  entry  condition  (M=3.5).  Both  baseline  conditions  were 
significantly  different  from  both  CAU  conditions.  Taken  together,  these  results 
indicate  that  the  reduction  in  MW  between  baseline  and  CAU  designs  was 
significantly  greater  for  Ascent  operations  as  compared  to  Entry  operations. 


63 


Interaction  Plot  for  Response 

Data  Means 

7 


6 

c 

i 

I  5 

4 

3 

Baseline  CAU 

Design 

Figure  31 .  Interaction  between  Design  and  Phase  of  Operations 

for  Mental  Workload 

To  identify  the  details  of  Design/Operations  interaction,  the  differences 
between  Baseline  and  CAU  designs  were  plotted  for  each  of  the  crew  positions 
(see  Figure  32).  Between  Baseline  and  CAU  designs,  Mission  Specialists 
appeared  to  show  a  lesser  reduction  in  MW  than  did  Pilots  and  Commanders. 

Post  hoc  comparisons  were  used  to  verify  the  significance  of  this 
interaction.  The  mean  workload  measure  for  the  baseline  Mission  Specialist 
position  (6.0)  did  not  significantly  differ  from  the  baseline  Pilot  position  (M=6.7). 
However,  the  CAU  Mission  Specialist  position  (M=4.0)  was  found  to  be 
significantly  different  from  the  CAU  Pilot  position  (M=3.2).  All  baseline  positions 
were  significantly  different  from  all  CAU  positions.  Taken  together,  these  results 
indicate  that  the  reduction  in  mental  workload  between  baseline  and  CAU 
designs  was  significantly  less  for  the  Mission  Specialist  position  compared  to 
Pilot  position. 
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Interaction  Plot  for  Response 
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Figure  32.  Interaction  between  Design  and  Crew  Position 

for  Mental  Workload 

2.  Situation  Awareness  of  T rajectory 

The  results  of  the  ANOVA  for  situation  awareness  of  trajectory  revealed  a 
significant  three-way  interaction  at  the  p<.01  level  (see  Table  18).  To  identify  the 
details  of  this  interaction,  the  data  were  divided  by  crew  position,  and  the 
changes  between  Baseline  and  CAU  designs  were  plotted  for  both  Ascent  and 
Entry  operations  (see  Figures  33-35).  The  plots  for  CDR  and  MS2  positions  both 
indicated  a  greater  increase  in  SA  for  ascent  phases  when  compared  to  entry 
phases.  The  plot  for  PLT  position,  by  contrast,  shows  an  equivalent  increase  in 
SA  for  entry  and  ascent  phases. 

Post  hoc  comparisons  were  used  to  verify  the  significance  of  this 
difference.  CAU  ascent  PLT  (M=8.6)  was  found  to  be  significantly  different  from 
CAU  entry  PLT  (M=7.8).  The  CAU  ascent/entry  pairs  for  PLT  and  MS2  positions 
were  not  found  to  be  significantly  different  from  each  other.  Baseline  entry 
results  for  all  three  crew  positions  were  found  to  be  significantly  different  than  the 


65 


corresponding  baseline  ascent  results.  Taken  together,  these  results  indicate 
that  in  terms  of  trajectory  SA,  pilots  responded  differently  to  the  CAU  upgrades 
than  did  commanders  and  mission  specialists.  Between  baseline  and  CAU 
designs,  commanders  and  mission  specialists  showed  a  greater  increase  in  SA 
during  ascent  phases  when  compared  to  entry  phases.  Pilots,  by  contrast, 
improved  at  a  more  uniform  rate  across  both  phases  of  flight. 

The  results  also  revealed  a  highly  significant  main  effect  for  Design.  This 
effect  was  far  greater  than  that  of  the  interaction,  and  is  therefore  worth 
addressing.  Figures  33,  34,  and  35,  clearly  reveal  that  the  CAU  design  greatly 
outperforms  the  Baseline  design  in  terms  of  situation  awareness  of  trajectory. 


Source  of  Variation 

Sum  of  Squares 

df 

Mean  Square 

F 

P 

Design 

8046.8 

1 

8046.8 

3282.95 

0.000 

Ops 

209.27 

1 

209.27 

85.38 

0.000 

Position 

30.40 

2 

15.20 

6.20 

0.002 

Design  x  Ops 

40.57 

1 

40.57 

16.55 

0.000 

Design  x  Position 

13.84 

2 

6.92 

2.82 

0.060 

Ops  x  Position 

1.69 

2 

0.84 

0.34 

0.709 

Design  x  Ops  x  Position 

23.73 

2 

11.86 

4.84 

0.008 

Error 

3897.23 

1590 

2.45 

Total 

14732.04 

1601 

Table  18.  ANOVA  of  Trajectory  Situation  Awareness 
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Interaction  Plot  for  CDR 
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Figure  33.  Trajectory  SA  Interaction  between  Design  and  Phase  of  Operations 

for  the  Commander  Crew  Position 
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Figure  34.  Trajectory  SA  Interaction  between  Design  and  Phase  of  Operations 

at  the  Pilot  Crew  Position 
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Interaction  Plot  for  MS2 

Data  Means 


Figure  35.  Trajectory  SA  Interaction  between  Design  and  Phase  of  Operations 

at  the  Mission  Specialist  Crew  Position 

3.  Situation  Awareness  of  Critical  Systems 

The  ANOVA  for  situation  awareness  of  critical  systems  did  not  reveal  a 
significant  three-way  interaction.  However,  a  significant  two-way  interaction  was 
found  between  Design  and  Operations  at  the  p<.01  level  (see  Table  19).  To 
identify  the  details  of  this  interaction,  the  differences  between  baseline  and  CAU 
designs  were  plotted  for  both  Ascent  and  Entry  operations  (see  Figure  36). 
Between  baseline  and  CAU  designs,  crews  appeared  to  show  a  greater  increase 
in  critical  system  SA  during  ascent  phases  when  compared  to  entry  phases. 

Post  hoc  comparisons  were  used  to  verify  the  significance  of  this 
difference.  Baseline  ascent  SA  was  found  to  be  significantly  different  from 
baseline  entry  SA.  CAU  ascent  and  entry  SA  were  not  found  to  be  significantly 
different  from  each  other.  Both  baseline  conditions  were  found  to  be  significantly 
different  than  both  ascent  conditions.  Taken  together,  these  results  indicated 
that  the  design  changes  affected  crew  SA  of  critical  systems  differently  in  ascent 


68 


operations  as  compared  to  entry  operations.  Specifically,  the  changes  yielded  a 
greater  increase  in  SA  during  Ascent  operations  than  during  Entry  operations. 

The  results  also  revealed  a  highly  significant  main  effect  for  Design.  This 
effect  was  far  greater  than  that  of  any  interaction.  As  seen  in  Figure  36,  the  CAU 
design  greatly  outperforms  the  Baseline  design  in  terms  of  situation  awareness 
of  critical  systems. 


Source  of  Variation 

Sum  of  Squares 

df 

Mean  Square 

F 

P 

Design 

10964.0 

1 

10964.0 

5361.26 

0.000 

Ops 

130.3 

1 

130.3 

63.70 

0.000 

Position 

48.4 

2 

24.2 

11.84 

0.000 

Design  x  Ops 

151.1 

1 

151.1 

73.87 

0.000 

Design  x  Position 

8.6 

2 

4.3 

2.11 

0.121 

Ops  x  Position 

8.1 

2 

4.0 

1.98 

0.139 

Design  x  Ops  x 

Position 

8.7 

2 

4.3 

2.13 

0.119 

Error 

6306.9 

3084 

2.0 

Total 

24262.1 

3095 

Table  19.  ANOVA  of  Critical  System  Situation  Awareness 
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Interaction  Plot  for  Response 
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Figure  36.  Critical  System  SA  Interaction  between  Design  and 

Phase  of  Operations 

4.  Situation  Awareness  of  Non-critical  Systems 

The  ANOVA  for  situation  awareness  of  non-critical  systems  did  not  reveal 
a  significant  three-way  interaction.  However,  a  significant  two-way  interaction 
was  found  between  Design  and  Operations  at  the  p<.01  level  (see  Table  20).  To 
identify  the  details  of  this  interaction,  the  differences  in  SA  between  baseline  and 
CAU  designs  were  plotted  for  both  Ascent  and  Entry  operations  (see  Figure  37). 
Between  baseline  and  CAU  designs,  crews  appeared  to  show  a  greater  increase 
in  non-critical  system  SA  during  ascent  phases  when  compared  to  entry  phases. 

Post  hoc  comparisons  were  used  to  verify  the  significance  of  this 
difference.  Baseline  ascent  SA  was  found  to  be  significantly  different  from 
baseline  entry  SA.  CAU  ascent  and  entry  SA  were  not  found  to  be  significantly 
different  from  each  other.  Both  baseline  conditions  were  found  to  be  significantly 
different  than  both  ascent  conditions.  Taken  together,  these  results  indicated 

that  the  design  changes  affected  crew  SA  of  non-critical  systems  differently  in 
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ascent  operations  as  compared  to  entry  operations.  Specifically,  the  changes 
yielded  a  greater  increase  in  SA  during  Ascent  operations  than  during  Entry 
operations. 

The  results  also  revealed  a  highly  significant  main  effect  for  Design.  This 
effect  was  far  greater  than  that  of  any  interaction.  Once  again,  the  CAU  design 
greatly  outperforms  the  Baseline  design  in  terms  of  situation  awareness  of  non- 
critical  systems  (see  Figure  37). 


Source  of  Variation 

Sum  of  Squares 

df 

Mean  Square 

F 

P 

Design 

5480.17 

1 

5480.17 

2586.7 

1 

0.000 

Ops 

35.63 

1 

35.63 

16.82 

0.000 

Position 

17.38 

2 

8.69 

4.10 

0.017 

Design  x  Ops 

18.33 

1 

18.33 

8.65 

0.003 

Design  x  Position 

11.30 

2 

5.65 

2.67 

0.070 

Ops  x  Position 

6.11 

2 

3.05 

1.44 

0.237 

Design  x  Ops  x  Position 

5.72 

2 

2.86 

1.35 

0.260 

Error 

2402.48 

1134 

2.12 

Total 

8137.70 

1145 

Table  20.  ANOVA  of  Non-Critical  Situation  Awareness 
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Figure  37.  Non-Critical  System  SA  Interaction  between  Design 

and  Phase  of  Operations 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

The  primary  purpose  of  the  study  was  to  complete  a  thorough  assessment 
of  the  Shuttle  CAU  design  improvements  and  to  develop  recommendations  for 
the  design  and  evaluation  of  future  systems  based  on  the  results  of  this 
assessment.  The  evaluation  began  with  a  critique  of  the  CAU  design 
modifications  in  terms  of  task  requirements,  panel  layout,  design  principles, 
hardware  attributes,  and  display  control.  Finally,  the  effectiveness  of  these 
modifications  was  examined  using  measures  of  MW  and  SA.  The  statistical 
analysis  focused  on  the  measures  with  the  highest  quality  data,  which  included 
Bedford  Workload  Rating  Scale,  and  Subjective  Situation  Awareness. 

For  every  performance  measure  examined,  a  significant  improvement  was 
found  between  the  baseline  and  CAU  designs  for  all  crew  positions  and  in  all 
phases  of  flight.  This  was  by  far  the  strongest  effect,  with  p-values  on  the  order 
of  10-30  for  every  measure.  The  analysis  also  revealed  that  these  improvements 
were  significantly  greater  for  ascent  operations  than  for  entry  in  all  but  one 
measure.  That  exception  was  found  in  the  Trajectory  Situation  Awareness 
measure,  wherein  pilots  improved  at  the  same  rate  in  both  ascent  and  entry 
operations. 

The  Ascent  H-SIT  display  is  a  development  that  is  applicable  exclusively 
during  ascent  scenarios  in  the  CAU  cockpit.  This  display  is  intended  to  improve 
ascent  trajectory  SA,  particularly  during  highly  dynamic  (and  attention¬ 
demanding)  abort  scenarios.  It  is  typically  visible  from  the  Commander  and 
Mission  Specialist  crew  positions,  but  outside  the  maximum  field  of  view  for 
pilots.  The  design  critique  also  noted  several  changes  in  design  that  apply 
throughout  all  conditions.  Namely,  the  changes  in  the  use  of  color,  the 
consolidation  of  information,  and  the  use  of  pictorial  representations  of  systems 
are  relevant  to  all  crew  positions  and  in  all  phases  of  flight. 
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B.  DISCUSSION 


1.  Research  Questions 

Three  research  questions  were  proposed  for  examination  in  this  thesis:  (1) 
Did  the  CAU  design  changes  lead  to  improvements  in  crew  mental  workload 
and/or  situation  awareness?;  (2)  If  mental  workload  and/or  situation  awareness 
improvements  exist,  do  they  vary  by  crew  station  or  phase  of  flight?;  and  (3)  Can 
crew  mental  workload,  and/or  situation  awareness  changes  be  traced  back  to 
specific  CAU  design  modifications? 

The  first  research  question  was  answered  definitively.  Highly  significant 
improvements  in  MW  and  SA  were  found  for  every  measure  between  the 
baseline  and  CAU  designs.  Furthermore,  the  design  critique  identified  several 
factors  in  the  CAU  design  changes,  including  use  of  color,  consolidation  of 
information,  and  use  of  pictorial  system  representations  that,  according  to  the 
literature,  should  have  resulted  in  reduced  MW  and  increased  SA.  It  is  therefore 
evident  that  the  CAU  design  changes  were  behind  the  significant  improvements 
that  were  identified. 

Answers  to  the  second  research  question  were  found  in  the  analyses  of 
variance.  In  particular,  significantly  greater  improvements  in  MW  and  SA  were 
found  for  ascent  operations  than  for  entry,  and  pilots  were  found  to  differ  from  the 
other  crew  positions  in  ascent  trajectory  SA  improvements.  These  results 
definitively  support  that  the  improvements  in  MW  and  varied  measurably  by  both 
crew  station  and  phase  of  flight. 

The  third  research  question  was  answered  by  synthesizing  the  results  of 
the  statistical  analysis  with  those  of  the  design  critique.  Overall  improvements 
between  baseline  and  CAU  designs  can  be  traced  to  the  use  of  color,  the 
consolidation  of  imagery,  and  the  pictorial  representation  of  systems.  However, 
the  relative  contributions  of  each  of  these  factors  cannot  be  determined. 
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The  difference  in  improvement  between  ascent  and  entry  operations, 
however,  points  to  the  ascent  H-SIT  display.  From  a  display  design  perspective, 
the  development  of  this  format  is  the  most  significant  change  that  affects  only 
ascent  operations.  This  determination  is  further  supported  by  the  observation 
that  astronauts  in  the  pilot  position  did  not  experience  the  same  improvement  as 
other  astronauts  in  measures  of  trajectory  SA.  Since  the  CAU  H-SIT  is  outside 
the  normal  field  of  view  from  the  pilot’s  seat  but  within  the  normal  field  of  view  for 
commanders  and  mission  specialists,  it  follows  that  pilots  would  not  experience 
the  same  benefits  as  the  other  positions. 

2.  Human  Systems  Integration  Domain  Applicability 

The  study  identified  a  connection  between  human  interface  design 
changes,  and  operator  MW  and  SA.  This  connection  is  directly  relevant  to  the 
HFE  domain.  The  lessons  learned  from  this  evaluation  are  broadly  applicable  to 
future  HFE  design  and  evaluation  activities. 

The  design  critique  also  identified  changes  in  task  requirements  between 
the  baseline  and  CAU  designs.  The  connection  between  changing  task 
demands  and  operator  performance  has  implications  in  the  training  domain.  This 
connection  demands  consideration  of  the  effect  of  design  on  the  knowledge, 
skills,  and  abilities  that  relate  to  the  operation  of  the  system. 

The  review  of  the  scientific  literature  identified  a  clear  connection  between 
operator  MW  and  SA,  and  system  safety  (e.g.,  Wickens,  2004;  Tsang  &  Vidulich, 
2006).  Consequently,  the  findings  of  this  thesis  that  relate  to  MW  and  SA  also 
represent  a  contribution  to  the  HSI  domain  of  Safety.  This  design  evaluation 
represents  one  way  in  which  safety  of  future  systems  can  be  improved. 

3.  Assessment  Criteria 

MW  and  SA  are  central  concepts  within  aerospace  human  factors 

research  (Wickens,  2002).  Because  of  this,  several  measures  of  MW  and  SA 

were  collected  during  the  course  of  this  evaluation.  MW  data  were  collected 
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using  both  BWRS  and  NASA-TLX  instruments,  and  SA  data  were  collected  using 
both  objective  and  subjective  measures.  Of  these,  BWRS  and  Subjective  SA 
measures  proved  to  be  the  most  beneficial  to  this  evaluation.  For  the  SA 
measures,  this  decision  was  primarily  based  on  the  greater  sampling  of 
subjective  data  vs.  objective  data.  For  the  MW  measures,  however,  TLX 
measures  showed  less  internal  consistency  than  BWRS  measures,  and  included 
possible  indications  of  scale  reversal  by  the  participants.  The  BWRS,  in 
comparison,  appeared  to  be  well  understood  by  all  participants. 

C.  CONCLUSIONS 

Collectively,  the  results  indicate  that  the  use  of  color,  consolidation  of 
information,  and  use  of  pictures  dramatically  improve  MW  and  SA.  These 
findings  are  consistent  with  the  potential  effects  described  in  Wickens  et  al. 
(2004).  However,  as  the  experiment  was  not  designed  distinguish  between  the 
effects  of  these  individual  changes,  no  conclusion  can  be  reached  about  their 
relative  contributions.  It  is  nonetheless  clear  that  these  design  principles  are 
highly  effective  when  used  together. 

The  evidence  does  support  a  more  specific  conclusion  regarding  the 
ascent  H-SIT  display.  The  difference  in  improvement  between  ascent  and  entry 
operations  indicates  that  the  predictive  aiding  provided  by  this  display  was 
particularly  beneficial.  The  observation  that  astronauts  who  could  not  see  the 
display  experienced  less  of  an  improvement  in  trajectory  SA  than  those  that 
could  further  supports  this  conclusion. 

D.  RECOMMENDATIONS 

Future  manned  space  systems  would  almost  certainly  benefit  from 
incorporating  the  design  principles  demonstrated  in  the  CAU.  The  use  of  a 
consistent  color  scheme,  task-centric  consolidation  of  information,  and  the  use  of 
pictorial  representations  of  complex  systems  are  clear,  positive  factors  in 
reducing  MW  and  increasing  SA  throughout  all  operations.  Furthermore, 
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predictive  aiding  displays  should  be  developed  for  use  during  all  dynamic  phases 
of  flight.  Because  of  the  ubiquity  of  the  underlying  principles,  these 
recommendations  can  be  further  generalized  to  manned  aircraft  as  well. 

In  evaluating  future  systems,  measurements  of  MW  and  SA  should 
include  the  BWRS  and  Subjective  SA.  Both  of  these  instruments  appeared  to  be 
well  understood  by  participants  and  provided  data  that  are  both  meaningful  and 
internally  consistent.  In  addition,  in  future  evaluations  it  would  be  of  tremendous 
benefit  to  inquire  of  the  participants  what  design  features  they  felt  had  the 
greatest  impact  on  overall  improvement.  Short  of  designing  an  experiment  that 
examines  design  features  in  isolation,  this  would  be  the  most  appropriate  way  to 
distinguish  specific  design  contributions  to  MW  and  SA. 
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APPENDIX  A.  EXAMPLES  OF  CAU  BENEFITS 


Note:  The  material  in  this  Appendix  has  been  condensed  and  adapted 
from  Evaluation  of  the  Space  Shuttle  Cockpit  Avionics  Upgrade  (CAU)  Displays 
by  J.W.  McCandless,  Revision  G,  dated  2004,  unpublished.  Used  with 
permission. 

A.  ELECTRICAL  BUS  FAILURE 

As  an  example  illustrating  the  differences  between  the  Baseline  and  CAU 
cockpits,  this  section  discusses  the  process  of  diagnosing  a  failure  of  the  Aft 
Power  Controller  (APC)  #6,  a  component  that  provides  power  to  a  number  of 
sensors,  heaters,  engine  gimbals,  valves,  and  other  redundant  systems.  This 
example  demonstrates  changes  that  primarily  relate  to  internal,  system  SA.  This 
process  is  discussed  for  both  Baseline  and  CAU  displays. 

1.  Baseline 

When  the  APC6  failure  occurs,  the  first  indicator  is  a  set  of  BFS  driven 
Fault  Messages  generated  by  the  Caution  and  Warning  System.  The  crew 
member  then  views  these  fault  messages  on  the  "BFS  FAULT"  summary  display 
(see  Figure  38).  The  first  of  the  three  messages,  "MPS  LH2/L02  ULL"  indicates 
a  problem  with  the  MPS.  The  failure  indicated  is  an  effect  of  the  loss  of  an  MPS 
sensor.  The  crewmember  must  process  several  such  indications  of  failures 
across  multiple  systems  in  order  to  infer  the  root  cause  of  the  problems. 
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1021/  /099  FAULT 
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BFS  000/00:  00:  00 

CRT  FAULT 

c/w 

GPC  TIME 

ID 

MPS  LH2/L02  ULL 

• 
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SMC  APU  SPD  LO  2 

* 

5  000/00:  01:  50 

SMD  THRM  APU 

5  000/00:  01:  50 

Figure  38.  Baseline  BFS  Fault  Summary  Display 

The  crewmember  has  further  insight  to  the  MPS  via  the  "BFS  GNC  SYS 
SUMM  1"  display  (Figure  39).  As  a  function  of  the  APC  6  failure,  the  "R  MPS 
ULL  P  LH2"  and  "L02"  values  show  'off-nominal'  values  of  "12.0"  with  an 
accompanying  down  arrow  for  "LH2"  and  zero  ("0")  for  "L02."  These  numbers 
indicate  how  much  pressure  is  available  in  the  external  propellant  storage  tanks 
to  supply  liquid  hydrogen  and  liquid  oxygen  to  each  main  engine.  If  the  "LH2" 
pressure  is  too  low,  the  engine  will  stop  running  from  lack  of  fuel.  With  no  other 
confirming  cues,  this  signature  looks  like  an  impending  loss  of  pressurization  in 
the  External  Tank  (ET). 

The  remaining  fault  messages,  "THRM  APU"  and  "APU  SPD  LO  2"  refer 
to  the  APU  system  that  provides  hydraulic  pressure  to  the  shuttle.  "THRM  APU" 
indicates  an  off  nominal  temperature  reading  for  one  of  the  APUs,  and  "APU 
SPD  LO  2"  indicates  a  low  turbine  speed  reading  for  APU  2.  The  "BFS  SM  SYS 
SUMM  2"  display  provides  more  specific  information,  showing  that  APU  2  has  an 
off-scale  low  "SPEED"  indicated  by  an  "L,”  and  APU  3  has  an  off  scale  low  "GG 
BED"  temperature,  also  indicated  by  an  "L"  (Figure  40). 
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Figure  40.  Baseline  BFS  SM  System  Summary  2  Display 


Taken  at  face  value,  the  signatures  appear  to  imply  the  loss  of  2  out  of  3 
APUs.  However,  in  this  instance  the  MPS  and  APU  signatures  are  the  result  of 
instrumentation  failures  caused  by  the  APC  6  failure.  To  determine  the  exact 
failure,  the  crewmember  must  utilize  secondary  cues  across  multiple  displays  to 
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confirm  that  the  SSMEs  and  APUs  are  actually  running  normally,  and  conclude 
that  these  indications  are  instrumentation  failures.  The  crewmember  must  then 
find  the  commonality  between  the  sensors  that  failed  simultaneously.  Given  the 
multitude  of  electrical  buses  and  sensors  on  the  vehicle,  this  "root  cause 
determination"  is  a  difficult  task. 

Inability  to  associate  these  signatures  into  a  "root  cause"  of  APC6  can 
have  serious  consequences.  A  crewmember  failing  to  identify  the  proper  failure 
means  that  they  have  low  situation  awareness  of  the  state  of  the  vehicle,  and  the 
correct  procedure  and  recovery  actions  are  not  taken.  This  can  lead  to  other 
equipment  failures  or  problems  later  in  the  flight. 

2.  CAU 

Looking  at  an  APC6  failure  in  CAU  cockpit,  we  find  that  the  Fault 
Messages  annunciated  for  the  failure  are  very  similar.  However,  unlike  the 
Baseline  "BFS  FAULT"  summary  display,  the  CAU  "Fault  Sum"  display  has  a 
direct  indication  of  a  problem  in  the  Electrical  Power  System  (EPS)  section  (see 
Figure  41).  The  red  "c"  in  the  lower  right  of  the  section  labeled  "EPS"  indicates  a 
sub-bus  failure  on  the  Main  C  electrical  bus.  The  "Fault  Sum"  also  shows  no 
indications  of  MPS  or  APU-related  failures.  The  red  "c"  directs  the  crewmember 
to  navigate  to  the  EPS  Summary  ("EPS  Sum")  display. 
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Figure  41 .  CAU  Fault  Sum  Display 

The  "EPS  Sum"  display  shows  a  layout  of  the  shuttle's  electrical  bus 
system  (see  Figure  42).  The  EPS  is  divided  into  three  Main  buses  (A,  B,  and  C) 
that  each  have  a  set  of  sub-buses.  APC6  is  a  sub-bus  of  Main  C.  The  "EPS 
Sum"  display  colors  the  APC6  box  and  interior  label  red,  indicating  that  the  APC6 
bus  has  failed.  By  directing  the  crews'  attention  to  the  appropriate  location  on  the 
display  via  color-coding,  the  source  of  the  failure  is  immediately  apparent, 
eliminating  the  need  to  perform  time-consuming  and  difficult  cognitive  operations 
to  determine  the  root  cause  of  the  failure.  Once  the  root  cause  is  known,  the 
crewmembers  can  reference  their  Flight  Data  File  (FDF)  to  determine  capabilities 
and  redundancies  lost  throughout  the  cockpit  and  take  the  appropriate 
recovery/reconfiguration  actions.  These  losses  are  also  displayed  to  the  crew  on 
the  appropriate  displays,  including  the  MPS  and  APU  displays. 
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Figure  42.  CAU  EPS  Summary  Display 


If  the  crewmember  looked  at  the  "MPS  Sum"  display  to  see  what  the 
SSME  impacts  were  to  the  APC6  failure,  he  or  she  would  see  that  the  Ullage 
pressure  ("H2  UN")  value  for  the  right  engine  has  been  replaced  with  the  missing 
data  symbol  'm'  in  cyan,  indicating  that  the  data  is  missing  (see  Figure  43). 
Since  the  APC6  failure  removed  power  from  this  sensor,  the  CAU  cockpit 
statuses  the  data  as  missing  rather  than  showing  invalid  data  as  in  the  MEDS 
cockpit.  Additionally,  the  CAU  display  shows  that  Helium  valve  A  on  the  Right 
SSME  is  failed  to  the  closed  portion  by  coloring  it  red.  This  valve  is  one  of  two 
redundant  valves  on  the  R  SSME  that  provides  helium  required  for  the  engine  to 
run.  This  failure  indication  is  very  valuable  in  letting  the  crewmember  know  not  to 
close  the  "B"  valve  at  any  time  in  the  future  while  the  engine  is  running. 
Otherwise,  the  engine  would  be  starved  of  helium  and  immediately  shutdown. 
An  engine  shutting  down  due  to  a  loss  of  helium  is  a  potentially  catastrophic 
condition.  The  "failed  closed"  status  of  the  helium  valve  is  not  readily  available  in 
the  MEDS  cockpit. 
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Figure  43.  CAU  MPS  Sum  Display 

B.  TRAJECTORY 

Another  example  illustrating  the  benefits  of  the  CAU  displays  concept  over 
the  Baseline  configuration  focuses  on  the  vehicle  trajectory.  This  example 
demonstrates  changes  that  primarily  relate  to  spatial/temporal  SA.  This  process 
is  discussed  for  both  Baseline  and  CAU  displays. 

1.  Baseline 

The  "H-SIT"  display  contains  an  overhead  picture  of  the  relationship  of  the 
shuttle  and  the  runway  (see  Figure  44).  In  other  words,  it  depicts  what  the 
shuttle  and  runway  would  look  like  if  viewed  from  an  altitude  above  the  shuttle 
looking  straight  down.  The  shuttle  symbol  in  the  lower  center  of  this  section  is  in 
a  fixed  position  on  the  display.  The  three  dots  in  front  of  the  shuttle  are  a 
prediction  of  where  the  shuttle  will  be  in  20,  40,  and  60  seconds  based  on 
parameters  including  speed  and  bank  angle.  The  runway  graphic  (the  circle  with 
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the  line  in  it)  represents  the  Heading  Alignment  Cone  (HAC),  and  the  direction  of 
the  runway.  The  shuttle  approaches  the  runway  and  makes  a  partial  circle 
approach  prior  to  lining  up  with  the  runway  and  landing.  The  runway  graphic  in 
this  display  example  shows  the  runway  off  to  the  left  of  the  nose  of  the  shuttle, 
and  the  shuttle  banking  left  (the  3  dots  arc  to  the  left)  to  approach  it.  Noticeably 
absent  from  this  display  is  any  indication  of  whether  the  shuttle  has  enough 
"energy"  to  make  the  runway.  In  other  words,  does  the  shuttle  currently  have 
enough  speed  and  altitude  to  fly  to  the  runway? 


Figure  44.  Baseline  Horizontal  Situation  Display 

With  Baseline  displays,  the  only  onboard  approximation  of  this  is  to  use 
range  (R)  and  altitude  (H)  data  from  the  ENTRY  ALPHA  cue  card  (see  Figure 
45).  Using  this  cue  card  along  with  knowledge  of  the  current  altitude  from  the 
displays,  the  crew  can  compare  their  current  range  to  the  expected  nominal 
range  on  the  cue  card.  The  crew  notes  their  current  (actual)  altitude  and  range 
(indicated  on  the  displays),  then  compares  the  range  on  the  card  for  their  given 
altitude  with  the  actual  range.  For  example,  if  the  displays  show  that  the  space 
shuttle  altitude  is  227K  feet,  the  crew  would  look  on  the  ENTRY  ALPHA  cue  card 
and  determine  that  they  should  be  1508  nautical  miles  from  the  landing  site.  If 
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the  crew's  actual  range  (shown  on  the  displays)  does  not  match  the  range  on  the 
card,  the  vehicle  is  either  high  or  low  energy.  To  continue  the  preceding 
example,  if  the  displays  show  the  actual  range  is  1397  nautical  miles,  the  crew 
would  know  they  are  in  a  high  energy  state.  However,  the  crew  cannot 
determine  whether  the  selected  site  is  still  achievable.  Additionally,  the  displays 
show  the  range  to  only  the  currently  selected  site  (not  alternate  sites).  If  energy 
to  the  selected  site  is  in  question,  the  crewmember  must  select  alternate  sites, 
and  then  evaluate  the  range  and  altitude 


ENTRY  ALPHA 


VR 

OCref 

R 

H  Href 

Rref 

25 

40 

4282  400 

24 

HI  40  LO 

2539 

247  -48  L79 

23 

43  40  37 

2114 

239  -65 

69 

22 

43  40  37 

1774 

234  -84 

63 

21 

43  40  37 

1508 

227  -106 

60 

20 

43  40  37 

1292  221  -123 

59 

19 

43  40  37 

1122 

215  -144 

59 

18 

43  40  37 

982 

209  -165 

60 

17 

43  40  37 

865 

203  -182 

60 

16 

43  40  37 

769 

197  -190 

61 

Figure  45.  Entry  Alpha  Cue  Card 


2.  CAU 

The  CAU  version  of  the  Entry  H-SIT  contains  a  similar  horizontal  situation 
representation  of  that  of  the  Baseline  version  of  the  display  (see  Figure  46).  The 
shuttle  symbol  is  placed  at  a  fixed  position  on  the  display,  and  the  magenta  line 
connects  the  shuttle  to  the  runway,  which  moves  based  on  its  horizontal 
relationship  to  the  shuttle.  In  the  figure  provided,  the  runway  is  currently  off  to 
the  left  of  the  shuttle  nose.  This  indicator  also  shows  that  the  shuttle  is  banked  to 
the  left  towards  the  site.  This  gives  the  crew  the  awareness  that  they  are  moving 
towards  the  site,  not  away  from  it. 
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Figure  46.  CAU  Entry  Horizontal  Situation  Display 

The  CAU  display  also  adds  some  key  pieces  of  situation  awareness  that 
are  not  available  on  the  Baseline  displays.  The  display  provides  an  indication  on 
when  the  shuttle  will  command  a  "roll  reversal."  During  the  course  of  entry,  the 
shuttle  does  not  fly  straight  at  the  runway,  but  rather  flies  a  series  of  left  and  right 
turns  banking  one  way  and  then  the  other  as  it  descends  through  the 
atmosphere.  While  the  shuttle  is  banked  left,  its  flight  path  turns  to  the  left  and 
the  runway  moves  off  to  the  right  of  the  display.  Eventually  the  shuttle  has  to 
reverse  roll  and  bank  to  the  right  to  avoid  the  runway  moving  too  far  off  to  the 
right  to  make  a  landing.  The  two  gray  lines  extending  from  the  shuttle  graphic 
(they  look  like  a  big  "V")  indicate  at  what  point  the  shuttle  will  command  a  roll 
reversal.  In  the  example  picture,  the  shuttle  is  banked  to  the  left  towards  the  site 
(the  runway  is  moving  to  the  right).  When  the  runway  moves  far  enough  to  the 
right,  the  magenta  line  will  cross  the  right  line  of  the  "V"  and  the  either  the 
autopilot  or  the  crewmember  (flying  manually)  will  reverse  the  bank  angle  and 
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turn  back  toward  the  runway.  A  countdown  timer  to  the  next  roll  reversal  is 
located  at  the  bottom  right  of  the  display.  The  display  example  indicates  that 
there  is  more  than  two  minutes  to  the  next  roll  reversal.  As  the  vehicle  gets 
closer,  the  timer  will  begin  counting  down  from  two  minutes  and  begin  flashing 
when  the  roll  reversal  commences.  Additionally,  the  display  also  provides  digital 
readouts  of  the  "AAz"  (delta  azimuth,  called  Del  Az)  which  is  the  angle  between 
the  shuttle's  current  velocity  vector  and  the  vector  to  the  landing  site  and  range 
("Rng").  These  two  pieces  of  data  were  available  in  Baseline  conditions  on 
another  display.  The  range  is  also  depicted  in  graphical  form  indicated  by  the 
"range  rings."  In  the  above  figure,  the  plot  shows  the  shuttle  approximately  2900 
nautical  miles  away  from  the  site. 

When  the  crew  is  flying  manually,  the  CAU  display  also  gives  the  crew  a 
flashing  alert  when  a  roll  reversal  is  approaching.  When  there  is  less  than  10 
seconds  to  a  roll  reversal,  a  "Roll  in  <10s"  alert/message  is  displayed  in  this 
section.  When  a  roll  reversal  needs  to  be  commanded  a  "Reverse  Roll"  alert  is 
displayed  in  its  place.  These  alerts  help  keep  the  crew  aware  and  help  avoid 
missing  or  drastically  delaying  a  roll  reversal  when  flying  manually.  Not 
performing  roll  reversals  promptly  could  potentially  result  in  not  achieving  landing 
on  a  runway. 

The  top-left  portion  of  this  display  also  contains  information  on  the 
shuttle's  current  energy  state.  This  is  a  completely  new  set  of  information  for  the 
crew.  This  section  is  comprised  mainly  of  an  "energy  footprint.”  This  footprint 
displays  information  on  what  runways  are  achievable  for  landing  based  on  the 
current  energy  state  (in  a  basic  sense,  current  velocity,  altitude  and  distance  from 
the  site).  The  center  region  (looks  like  an  upside  down  home  plate)  is  the 
nominal  energy  region.  Runways  that  appear  in  this  section  are  achievable  using 
nominal  guidance  commands.  The  regions  outside  of  the  nominal  energy  region 
are  low  energy  regions.  The  sites  located  in  these  areas  of  the  footprint  are 
achievable  only  when  the  shuttle  flies  a  profile  designed  to  "stretch"  its  energy 
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out  with  special  flying  techniques  that  minimize  the  energy  loss  during  the  entry 
profile.  Sites  that  are  in  the  nominal  footprint  are  colored  white,  which  is  the 
nominal  color.  Sites  that  are  in  the  low  energy  footprint  are  colored  yellow,  which 
indicates  that  they  are  an  off-nominal  condition.  Sites  that  are  outside  of  the 
nominal  and  low  energy  regions  are  colored  red.  Red  and  yellow  are  the  warning 
and  caution  colors,  respectively.  In  the  example  above,  both  "KSC"  and  "NKT" 
runways  are  available  in  the  nominal  energy  footprint,  and  "YHZ"  (colored  yellow) 
is  available  through  the  low  energy  techniques.  The  CAU  display  greatly 
simplifies  this  energy  assessment  process,  allowing  the  crew  to  make  quick  and 
correct  decisions  on  which  runway  to  fly  towards.  The  logic  used  in  the  energy 
evaluation  is  much  more  precise  than  the  cue  card  method,  and  allows  the  crew 
to  assess  up  to  three  sites  simultaneously.  The  remaining  portions  of  the 
Baseline  display  were  moved  to  other  CAU  navigation  displays  to  fit  in  with  the 
task  oriented  design  concept. 

When  the  shuttle  gets  closer  to  the  runway  and  enters  the  Terminal  Area 
Energy  Management  (TAEM)  phase  of  flight  (below  Mach  2.5  for  a  nominal 
entry),  the  CAU  display  tailors  itself  to  flying  in  that  regime.  The  energy  state 
footprint  is  no  longer  needed,  and  the  runway  representation  is  swapped  to  show 
the  HAC.  Thus,  the  display  provides  the  most  important  information  when  it  is 
needed. 
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APPENDIX  B.  CAU  COLOR  STANDARDS 


The  upgraded  CAU  display  formats  use  color  to  enable  the  crew  to 
differentiate  between  the  various  classes  of  information,  especially  during  off- 
nominal  conditions.  Colors  were  primarily  chosen  based  on  usability  principles, 
but  were  subject  to  hardware  limitations.  The  color  scheme  used  throughout  the 
CAU  display  formats  is  presented  in  Table  21 . 


Color 

Associated  Elements 

Dark  Blue 

Background 

Dark  Grey 

Separator  lines,  non-critical  elements 

Light  Blue/ 

Grey 

Display  labels 

White 

Nominal  information 

Magenta 

Commanded  messages  which  are  critical  for  crew  to  read 

Light  Green 

Display  title,  navigation  elements,  highlighting 

Red 

warning 

Yellow 

caution 

Orange 

Disagreement  between  primary  and  backup  software 

Cyan 

Data  unavailable 

Table  21 

CAU  Color  Standards  (After  McCandless,  2004) 
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